"Fossies" - the Fresh Open Source Software Archive

Member "freezer-10.0.0/doc/source/index.rst" (14 Apr 2021, 12005 Bytes) of package /linux/misc/openstack/freezer-10.0.0.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) reStructured Text source code syntax highlighting (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file.

    1 ==================================
    2 Welcome to Freezer's documentation
    3 ==================================
    4 
    5 Freezer is a distributed backup restore and disaster recovery as a service
    6 platform. It is designed to be multi OS (Linux, Windows, OSX, BSD), focused
    7 on providing efficiency and flexibility for block based backups, file based
    8 incremental backups, point-in-time actions, jobs synchronization (i.e. backup
    9 synchronization over multiple nodes) and many other features. It is aimed at
   10 being useful for all environments, including large ephemeral Clouds.
   11 
   12 
   13 Key Features
   14 ============
   15 
   16 -  Backup your file system using point in time snapshot
   17 -  Strong encryption supported: AES-256-CFB
   18 -  Backup your file system tree directly (without volume snapshot)
   19 -  Backup your journalled MongoDB directory tree using LVM snapshot to swift
   20 -  Backup MySQL DB with LVM snapshot
   21 -  Restore your data from a specific date automatically to your file system
   22 -  Low storage consumption as the backup are uploaded as a stream
   23 -  Flexible backup policy (incremental and differential)
   24 -  Data is archived in GNU Tar format for file based incremental
   25 -  Block based backup support (rsync)
   26 -  Multiple compression algorithm support (zlib, bzip2, xz)
   27 -  Remove old backup automatically according to the provided parameters
   28 -  Multiple storage media support (Swift, local file system, ssh)
   29 -  Flush kernel buffered memory to disk
   30 -  Multi platform (Linux, Windows, \*BSD, OSX)
   31 -  Manage multiple jobs (i.e. multiple backups on the same node)
   32 -  Synchronize backups and restore on multiple nodes
   33 -  Web user interface integrated with OpenStack Horizon
   34 -  Can execute scripts/commands before or after a job execution
   35 
   36 
   37 Low Resources Requirement
   38 >>>>>>>>>>>>>>>>>>>>>>>>>
   39 
   40 Freezer is designed to reduce to the minimum I/O, CPU and Memory Usage.
   41 This is achieved by generating a data stream from tar (for archiving)
   42 and gzip (for compressing). Freezer segments the stream in a configurable
   43 chunk size (with the option --max-seg-size). The default segment size is
   44 64MB, so it can be safely stored in memory, encrypted if the key is
   45 provided, and uploaded to Swift as a segment.
   46 
   47 Multiple segments are sequentially uploaded using the Swift Manifest.
   48 All the segments are uploaded first, and then the Manifest file is
   49 uploaded too, so the data segments cannot be accessed directly. This
   50 ensures data consistency.
   51 
   52 By keeping the segments small, in-memory, I/O usage is reduced. Also as
   53 there's no need to store locally the final compressed archive
   54 (tar-gziped), no additional or dedicated storage is required for the
   55 backup execution. The only additional storage needed is the LVM snapshot
   56 size (set by default at 5GB). The lvm snapshot size can be set with the
   57 option --lvm-snapsize. It is important to not specify a too small snapshot
   58 size, because in case a quantity of data is being written to the source
   59 volume and consequently the lvm snapshot is filled up, then the data is
   60 corrupted.
   61 
   62 If more memory is available for the backup process, the maximum
   63 segment size can be increased. This will speed up the process. Please
   64 note that the segments must be smaller then 5GB, since that is the maximum
   65 object size in the Swift server.
   66 
   67 On the other hand, if a server has small memory availability, the
   68 --max-seg-size option can be set to lower values. The unit of this
   69 option is in bytes.
   70 
   71 How the Incremental Works
   72 >>>>>>>>>>>>>>>>>>>>>>>>>
   73 
   74 The incremental backups is one of the most crucial features. The
   75 following basic logic happens when Freezer executes:
   76 
   77 1) Freezer starts the execution and checks if the provided backup name for
   78    the current node already exists in Swift.
   79 
   80 2) If the backup exists, then the Manifest file is retrieved. This is
   81    important as the Manifest file contains the information of the
   82    previous Freezer execution.
   83 
   84 
   85 Freezer Components
   86 ==================
   87 
   88 +-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
   89 | Component         | Description                                                                                                                                    |
   90 +===================+================================================================================================================================================+
   91 | Freezer Web UI    | Web interface that interacts with the Freezer API to configure and change settings.                                                            |
   92 |                   | It provides most of the features from the Freezer Agent CLI, advanced scheduler settings such as multi-node backup synchronization,            |
   93 |                   | metrics, and reporting.                                                                                                                        |
   94 +-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
   95 | Freezer Scheduler | A client side component, running on the node where the data backup is to be executed.                                                          |
   96 |                   | It consists of a daemon that retrieves the data from the freezer API and executes jobs (i.e. backups, restore, admin actions, info actions,    |
   97 |                   | pre and/or post job scripts) by running the Freezer Agent.                                                                                     |
   98 |                   | The metrics and exit codes returned by the freezer agent are captured and sent to the Freezer API.                                             |
   99 |                   | The scheduler manages the execution and synchronization of multiple jobs executed on a single or multiple nodes.                               |
  100 |                   | The status of the execution of all the nodes is saved through the API.                                                                         |
  101 |                   | The Freezer scheduler takes care of uploading jobs to the API by reading job files on the file system.                                         |
  102 |                   | It also has its own configuration file where job session or other settings like the freezer API polling interval can be configured.            |
  103 |                   | The Freezer scheduler manages jobs, for more information about jobs please refer to: freezer_api/README.rst under JOB the sections             |
  104 +-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
  105 | Freezer Agent     | Multiprocessing Python software that runs on the client side, where the data backup is to be executed.                                         |
  106 |                   | It can be executed standalone or by the Freezer Scheduler.                                                                                     |
  107 |                   | The Freezer Agent provides a flexible way to execute backup, restore and other actions on a running system.                                    |
  108 |                   | In order to provide flexibility in terms of data integrity, speed, performance, resources usage, etc the freezer agent offers a                |
  109 |                   | wide range of options to execute optimized backup according to the available resources as:                                                     |
  110 |                   | # Segments size (the amount of memory used)                                                                                                    |
  111 |                   | # Queues size (optimize backups where I/O, bandwidth, memory or CPU is a constraint)                                                           |
  112 |                   | # I/O Affinity and process priority (it can be used with real time I/O and maximum user level process priority)                                |
  113 |                   | # Bandwidth limitation                                                                                                                         |
  114 |                   | # Client side Encryption (AES-256-CFB)                                                                                                         |
  115 |                   | # Compression (multiple algorithms supported as zlib, bzip2, xz/lzma)                                                                          |
  116 |                   | # Parallel upload to pluggable storage media (i.e., upload backup to swift and to a remote node by ssh,                                        |
  117 |                   |   or upload to two or more independent swift instances with different credentials, etc)                                                        |
  118 |                   | # Execute file based incremental (like tar), block based incremental (like rsync algorithm) and differential based backup and restore          |
  119 |                   | # Multi-platform as it can be run on Linux, Windows, \*BSD and OSX                                                                             |
  120 |                   | # Automatic removal of old backups                                                                                                             |
  121 +-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
  122 | Freezer API       | The API is used to store and provide metadata to the Freezer Web UI and to the Freezer Scheduler.                                              |
  123 |                   | Also the API is used to store session information for multi node backup synchronization. No workload data is stored in the API.                |
  124 |                   | For more information to the API please refer to: freezer_api/README.rst                                                                        |
  125 +-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
  126 | DB                | Backend used by the API to store and retrieve metrics, metadata sessions information, job status, etc.                                         |
  127 |                   | The DB support Elasticserch and Sqlalchemy.                                                                                                    |
  128 +-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
  129 
  130 
  131 Architecture
  132 ============
  133 
  134 Freezer may seem complex at first look. But Following diagrams will give
  135 you better understanding of the architecture:
  136 
  137 Service Architecture
  138 >>>>>>>>>>>>>>>>>>>>
  139 
  140 .. image:: images/admin/Service_Architecture_02.png
  141    :width: 640 px
  142 
  143 Agent Backup Work Flow with API
  144 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  145 
  146 .. image:: images/admin/freezer_agent_backup_api.png
  147    :width: 640 px
  148 
  149 Agent Backup Without API
  150 >>>>>>>>>>>>>>>>>>>>>>>>
  151 
  152 .. image:: images/admin/freezer_agent_backup.png
  153    :width: 640 px
  154 
  155 Freezer Scheduler with API
  156 >>>>>>>>>>>>>>>>>>>>>>>>>>
  157 
  158 .. image:: images/admin/freezer_scheduler_api.png
  159    :width: 640 px
  160 
  161 Freezer Job Session
  162 >>>>>>>>>>>>>>>>>>>
  163 
  164 .. image:: images/admin/job_session.png
  165    :width: 640 px
  166 
  167 Freezer Dashboard
  168 >>>>>>>>>>>>>>>>>
  169 
  170 .. image:: images/admin/freezer_dashboard.png
  171    :width: 640 px
  172 
  173 How to Scale
  174 >>>>>>>>>>>>
  175 
  176 .. image:: images/admin/freezer_scheduler_api_scale.png
  177    :width: 640 px
  178 
  179 
  180 
  181 How to Use
  182 ==========
  183 
  184 Install Guides
  185 >>>>>>>>>>>>>>
  186 .. toctree::
  187    :maxdepth: 2
  188 
  189    install/index
  190 
  191 
  192 User Guides
  193 >>>>>>>>>>>
  194 .. toctree::
  195    :maxdepth: 2
  196 
  197    user/index
  198    cli/index
  199 
  200 Admin Guides
  201 >>>>>>>>>>>>
  202 .. toctree::
  203    :maxdepth: 2
  204 
  205    admin/index
  206 
  207 Dev Guides
  208 >>>>>>>>>>
  209 .. toctree::
  210    :maxdepth: 2
  211 
  212    contributor/index
  213    reference/index