"Fossies" - the Fresh Open Source Software Archive

Member "bareos-Release-20.0.3/core/README.droplet" (14 Sep 2021, 5624 Bytes) of package /linux/misc/bareos-Release-20.0.3.tar.gz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file.

    1 Using droplet S3 as a backingstore for backups.
    2 
    3 The droplet S3 storage backend writes chunks of data in an S3 bucket.
    4 
    5 For this you need to install the the bareos-storage-droplet packages which contains
    6 the libbareossd-chunked*.so  and  libbareossd-droplet*.so shared objects and the droplet storage backend which implements a dynamic loaded storage backend.
    7 
    8 In the following example all the backup data is placed in the "bareos-backup" bucket on the defined S3 storage.
    9 A volume is a sub-directory in the defined bucket, and every chunk is placed in the Volume directory with the filename 0000-9999 and a size that is defined in the chunksize.
   10 
   11 The droplet S3 can only be used with virtual-hosted-style buckets like http://<bucket>.<s3_server>/object
   12 Path-style buckets are not supported when using the droplet S3.
   13 
   14 On the Storage Daemon the following configuration is needed.
   15 Example bareos-sd.d/device/S3_ObjectStorage.conf file:
   16 
   17 Device {
   18   Name = S3_ObjectStorage
   19   Media Type = S3_Object1
   20   Archive Device = S3 Object Storage
   21 
   22   #
   23   # Device Options:
   24   #    profile= - Droplet profile path, e.g. /etc/bareos/bareos-sd.d/droplet/droplet.profile
   25   #    location= - AWS location (e.g. us-east etc.). Optional.
   26   #    acl= - Canned ACL
   27   #    storageclass= - Storage Class to use.
   28   #    bucket= - Bucket to store objects in.
   29   #    chunksize= - Size of Volume Chunks (minimum = default = 10 Mb)
   30   #    iothreads= - Number of IO-threads to use for upload (use blocking uploads if not defined)
   31   #    ioslots= - Number of IO-slots per IO-thread (0-255, default 10)
   32   #    retries= - Number of retires if a write fails (0-255, default = 0, which means unlimited retries)
   33   #    mmap - Use mmap to allocate Chunk memory instead of malloc().
   34   #
   35 
   36   # testing:
   37   Device Options = "profile=/etc/bareos/bareos-sd.d/droplet/droplet.profile,bucket=bareos-bucket,chunksize=100M,iothreads=0,retries=1"
   38 
   39   # performance:
   40   #Device Options = "profile=/etc/bareos/bareos-sd.d/droplet/droplet.profile,bucket=bareos-bucket,chunksize=100M"
   41 
   42   Device Type = droplet
   43   LabelMedia = yes                    # lets Bareos label unlabeled media
   44   Random Access = yes
   45   AutomaticMount = yes                # when device opened, read it
   46   RemovableMedia = no
   47   AlwaysOpen = no
   48   Description = "S3 device"
   49   Maximum File Size = 500M            # 500 MB (allows for seeking to small portions of the Volume)
   50   Maximum Concurrent Jobs = 1
   51   Maximum Spool Size = 15000M
   52 }
   53 
   54 The droplet.profile file holds the credentials for S3 storage
   55 Example /etc/bareos/bareos-sd.d/droplet/droplet.profile file:
   56 
   57 Make sure the file is only readable for bareos, credentials for S3 are listed here.
   58 
   59 Config options profile:
   60 
   61 use_https = True
   62 host = <FQDN for S3>
   63 access_key = <S3 access key>
   64 secret_key = <S3 secret key>
   65 pricing_dir = ""
   66 backend = s3
   67 aws_auth_sign_version = 2
   68 
   69 If the pricing_dir is not empty,
   70 it will create an <profile_directory>/droplet.csv file which will record all S3 operations.
   71 See the code at https://github.com/bareos/Droplet/blob/bareos-master/libdroplet/src/pricing.c for an explanation.
   72 
   73 The parameter "aws_auth_sign_version = 2" is for the connection to a CEPH S3 gateway.
   74 For use with AWS S3 the aws_auth_sign_version, must be set to "4".
   75 
   76 On the Director you connect to the Storage Device with the following configuration
   77 Example bareos-dir.d/storage/S3_1-00.conf file:
   78 
   79 Storage {
   80   Name = S3_Object
   81   Address  = "Replace this by the Bareos Storage Daemon FQDN or IP address"
   82   Password = "Replace this by the Bareos Storage Daemon director password"
   83   Device = S3_ObjectStorage
   84   Media Type = S3_Object1
   85 }
   86 
   87 
   88 Troubleshooting
   89 ===============
   90 
   91 S3 Backend Unreachable
   92 ----------------------
   93 
   94 The droplet device can run in two modes:
   95   * direct writing (iothreads  = 0)
   96   * cached writing (iothreads >= 1)
   97 
   98 If iothreads >= 1, retries = 0 (unlimited retries) and the droplet backend (e.g. S3 storage) is not available, a job will continue running until the backend problem is fixed.
   99 If this is the case and the job is canceled, it will only be canceled on the Director. It continues running on the Storage Daemon, until the S3 backend is available again or the Storage Daemon itself is restarted.
  100 
  101 If iothreads >= 1, retries != 0 and the droplet backend (e.g. S3 storage) is not available, write operation will be silently discarded after the specified number of retries.
  102 *Don't use this combination of options*.
  103 
  104 Caching when S3 backend is not available:
  105 This behaviour have not changed, but I fear problems can arise, if the backend is not available and all write operations are stored in memory.
  106 
  107 The status of the cache can be determined with the "status storage=..." command.
  108 
  109 
  110 Pending IO chunks (and inflight chunks):
  111 ```
  112 ...
  113 Device "S3_ObjectStorage" (S3) is mounted with:
  114     Volume:      Full-0085
  115     Pool:        Full
  116     Media type:  S3_Object1
  117 Backend connection is working.
  118 Inflight chunks: 2
  119 Pending IO flush requests:
  120    /Full-0085/0002 - 10485760 (try=0)
  121    /Full-0085/0003 - 10485760 (try=0)
  122    /Full-0085/0004 - 10485760 (try=0)
  123 ...
  124 Attached Jobs: 175
  125 ...
  126 ```
  127 
  128 If try > 0, problems did already occur. The system will continue retrying.
  129 
  130 
  131 Status without pending IO chunks:
  132 ```
  133 Device "S3_ObjectStorage" (S3) is mounted with:
  134     Volume:      Full-0084
  135     Pool:        Full
  136     Media type:  S3_Object1
  137 Backend connection is working.
  138 No Pending IO flush requests.
  139 Configured device capabilities:
  140   EOF BSR BSF FSR FSF EOM !REM RACCESS AUTOMOUNT LABEL !ANONVOLS !ALWAYSOPEN
  141 Device state:
  142   OPENED !TAPE LABEL !MALLOC APPEND !READ EOT !WEOT !EOF !NEXTVOL !SHORT MOUNTED
  143   num_writers=0 reserves=0 block=8
  144 Attached Jobs:
  145 ```