"Fossies" - the Fresh Open Source Software Archive  

Source code changes of the file "docs/manuals/source/TasksAndConcepts/Plugins.rst" between
bareos-Release-20.0.0.tar.gz and bareos-Release-20.0.1.tar.gz

About: Bareos is a network based backup solution offering many different backup strategies (a fork of "Bacula").

Plugins.rst  (bareos-Release-20.0.0):Plugins.rst  (bareos-Release-20.0.1)
skipping to change at line 280 skipping to change at line 280
already a lot of things required by Python 3. already a lot of things required by Python 3.
For more details see `Python 3 Porting Guide, strings chapter: <https://po rtingguide.readthedocs.io/en/latest/strings.html>`_ For more details see `Python 3 Porting Guide, strings chapter: <https://po rtingguide.readthedocs.io/en/latest/strings.html>`_
Switching to Python 3 Switching to Python 3
^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^
Switching to use the Python 3 plugin, the following needs to be changed: Switching to use the Python 3 plugin, the following needs to be changed:
* Set `Plugin Names = "python3"` to make sure the Python3 plugin is loaded. * Set `Plugin Names = "python3"` to make sure the Python3 plugin is loaded.
* Adapt the Plugin setting in the fileset to use Python3: `Plugin = "python3:m odule_path ...` * Adapt the Plugin setting in the fileset to use Python3: `Plugin = "python3:m odule_path ...`
.. warning::
It is not possible to load the python2 and python3 plugins at the same time.
Recovering old backups Recovering old backups
^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^
When doing backups, the plugin parameter string is stored into the backup stream . When doing backups, the plugin parameter string is stored into the backup stream .
During restore, this string is used to determine the plugin that will handle thi s During restore, this string is used to determine the plugin that will handle thi s
data. data.
To be able to restore backups created with Python plugins using the To be able to restore backups created with Python plugins using the
**python3-fd** plugin that were created using the **python-fd** plugin, **python3-fd** plugin that were created using the **python-fd** plugin,
the code determining the plugin that will handle the data also matches for the code determining the plugin that will handle the data also matches for
the basename of the current available plugins without the last character. the basename of the current available plugins without the last character.
skipping to change at line 375 skipping to change at line 379
.. code-block:: shell .. code-block:: shell
writer=sh -c 'cat >/var/tmp/bpipe.data' writer=sh -c 'cat >/var/tmp/bpipe.data'
Please note that the two items above describing the "reader" and "writer", these programs are "executed" by Bareos, which means there is no shell interpretation of any command line arguments you might use. If you want to use shell character s (redirection of input or output, ...), then we recommend that you put your com mand or commands in a shell script and execute the script. In addition if you ba ckup a file with reader program, when running the writer program during the rest ore, Bareos will not Please note that the two items above describing the "reader" and "writer", these programs are "executed" by Bareos, which means there is no shell interpretation of any command line arguments you might use. If you want to use shell character s (redirection of input or output, ...), then we recommend that you put your com mand or commands in a shell script and execute the script. In addition if you ba ckup a file with reader program, when running the writer program during the rest ore, Bareos will not
automatically create the path to the file. Either the path must exist, or you mu st explicitly do so with your command or in a shell script. automatically create the path to the file. Either the path must exist, or you mu st explicitly do so with your command or in a shell script.
See the examples about :ref:`backup-postgresql` and :ref:`backup-mysql`. See the examples about :ref:`backup-postgresql` and :ref:`backup-mysql`.
PGSQL Plugin
~~~~~~~~~~~~
See chapter :ref:`backup-postgresql-plugin`.
MySQL Plugin MySQL Plugin
~~~~~~~~~~~~ ~~~~~~~~~~~~
See the chapters :ref:`backup-mysql-XtraBackup` and :ref:`backup-mysql-python`. See the chapters :ref:`backup-mysql-XtraBackup` and :ref:`backup-mysql-python`.
MSSQL Plugin MSSQL Plugin
~~~~~~~~~~~~ ~~~~~~~~~~~~
See chapter :ref:`MSSQL`. See chapter :ref:`MSSQL`.
LDAP Plugin LDAP Plugin
~~~~~~~~~~~ ~~~~~~~~~~~
:index:`\ <single: Plugin; ldap>`\ :index:`\ <single: Plugin; ldap>`\
.. deprecated:: 20.0.0
This plugin is intended to backup (and restore) the contents of a LDAP server. I t uses normal LDAP operation for this. The package **bareos-filedaemon-ldap-pyth on-plugin** (:sinceVersion:`15.2.0: LDAP Plugin`) contains an example configurat ion file, that must be adapted to your environment. This plugin is intended to backup (and restore) the contents of a LDAP server. I t uses normal LDAP operation for this. The package **bareos-filedaemon-ldap-pyth on-plugin** (:sinceVersion:`15.2.0: LDAP Plugin`) contains an example configurat ion file, that must be adapted to your environment.
Cephfs Plugin Cephfs Plugin
~~~~~~~~~~~~~ ~~~~~~~~~~~~~
:index:`\ <single: Plugin; ceph; cephfs>`\ :index:`\ <single: Ceph; Cephfs Plug in>`\ :index:`\ <single: Plugin; ceph; cephfs>`\ :index:`\ <single: Ceph; Cephfs Plug in>`\
Opposite to the :ref:`Rados Backend <SdBackendRados>` that is used to store data on a CEPH Object Store, this plugin is intended to backup a CEPH Object Store v ia the Cephfs interface to other media. The package **bareos-filedaemon-ceph-plu gin** (:sinceVersion:`15.2.0: Cephfs Plugin`) contains an example configuration file, that must be adapted to your environment. Opposite to the :ref:`Rados Backend <SdBackendRados>` that is used to store data on a CEPH Object Store, this plugin is intended to backup a CEPH Object Store v ia the Cephfs interface to other media. The package **bareos-filedaemon-ceph-plu gin** (:sinceVersion:`15.2.0: Cephfs Plugin`) contains an example configuration file, that must be adapted to your environment.
Rados Plugin Rados Plugin
skipping to change at line 1633 skipping to change at line 1634
job_message_after_each_number_of_objects job_message_after_each_number_of_objects
When running a backup, put a jobmessage after each count of "job_message_afte r_number_of_objects" When running a backup, put a jobmessage after each count of "job_message_afte r_number_of_objects"
to the joblog or no message if parameter equals 0; default is 100. to the joblog or no message if parameter equals 0; default is 100.
.. _PerconaXtrabackupPlugin: .. _PerconaXtrabackupPlugin:
.. _backup-mysql-XtraBackup: .. _backup-mysql-XtraBackup:
Percona XtraBackup Plugin Percona XtraBackup Plugin
~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~
:index:`\ <single: Plugin; MySQL Backup>` .. index::
:index:`\ <single: Percona XtraBackup>` single: Plugin; MySQL Backup
:index:`\ <single: XtraBackup>` single: Percona XtraBackup
:index:`\ <single: Plugin; MariaDB Backup>` single: XtraBackup
single: Plugin; MariaDB Backup
This plugin uses Perconas XtraBackup tool, to make full and incremental backups of Mysql / MariaDB databases. This plugin uses Perconas XtraBackup tool, to make full and incremental backups of |mysql| databases.
The key features of XtraBackup are: The key features of XtraBackup are:
- Incremental backups - Incremental backups
- Backups that complete quickly and reliably - Backups that complete quickly and reliably
- Uninterrupted transaction processing during backups - Uninterrupted transaction processing during backups
- Savings on disk space and network bandwidth - Savings on disk space and network bandwidth
- Higher uptime due to faster restore time - Higher uptime due to faster restore time
Incremental backups only work for INNODB tables, when using MYISAM, only full ba ckups can be created. Incremental backups only work for INNODB tables, when using MYISAM, only full ba ckups can be created.
Prerequisites Prerequisites
^^^^^^^^^^^^^ ^^^^^^^^^^^^^
Install the XtraBackup tool from Percona. Documentation and packages are availab le here: https://www.percona.com/software/mysql-database/percona-XtraBackup. The plugin was successfully tested with XtraBackup versions 2.3.5 and 2.4.4. Install the XtraBackup tool from Percona. Documentation and packages are availab le here: https://www.percona.com/software/mysql-database/percona-XtraBackup. The plugin was successfully tested with XtraBackup versions 2.3.5 and 2.4.4.
As it is a Python plugin, it will also require to have the package **bareos-file
daemon-python-plugin** installed on the |fd|, where you run it.
For authentication the :file:`.mycnf` file of the user running the |fd| is used. Before proceeding, make sure that XtraBackup can connect to the database and cr eate backups. For authentication the :file:`.mycnf` file of the user running the |fd| is used. Before proceeding, make sure that XtraBackup can connect to the database and cr eate backups.
Installation Installation
^^^^^^^^^^^^ ^^^^^^^^^^^^
Make sure you have met the prerequisites, after that install the package **bareo s-filedaemon-percona_XtraBackup-python-plugin**. Make sure you have met the prerequisites, after that install the package **bareo s-filedaemon-percona_XtraBackup-python-plugin**.
Configuration Configuration
^^^^^^^^^^^^^ ^^^^^^^^^^^^^
skipping to change at line 1749 skipping to change at line 1749
Troubleshooting Troubleshooting
''''''''''''''' '''''''''''''''
If things don't work as expected, make sure that If things don't work as expected, make sure that
- the |fd| (FD) works in general, so that you can make simple file backups and r estores - the |fd| (FD) works in general, so that you can make simple file backups and r estores
- the Bareos FD Python plugins work in general, try one of - the Bareos FD Python plugins work in general, try one of
the shipped simple sample plugins the shipped simple sample plugins
- Make sure *XtraBackup* works as user root, MySQL access needs to be - Make sure *XtraBackup* works as user root, MySQL access needs to be
configured properly configured properly
.. _plugin-postgresql-fd:
PostgreSQL Plugin PostgreSQL Plugin
~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~
:index:`\ <single: Plugin; PostgreSQL Backup>` .. index::
single: Plugin; PostgreSQL Backup
The PostgreSQL plugin supports an online (Hot) backup of database files and data
base transaction logs (WAL) archiving. With online database and transaction logs
the backup plugin can perform Point-In-Time-Restore up to a single selected tra
nsaction or date/time.
This plugin uses the standard API |postgresql| backup routines based on *pg_sta rt_backup()* and *pg_stop_backup()*. This plugin uses the standard API |postgresql| backup routines based on *pg_sta rt_backup()* and *pg_stop_backup()*.
The key features are: The key features are:
* Incremental backups * Incremental backups
* Point in time recovery * Point in time recovery
* Backups that complete quickly and reliably * Backups that complete quickly and reliably
* Uninterrupted transaction processing during backups * Uninterrupted transaction processing during backups
* Savings on disk space and network bandwidth * Savings on disk space and network bandwidth
* Higher uptime due to faster restore time * Higher uptime due to faster restore time
Requires |postgresql| Version 9.x or newer.
Concept Concept
^^^^^^^ ^^^^^^^
Please make sure to read the |postgresql| documentation about the backup and res tore process: https://www.postgresql.org/docs/current/continuous-archiving.html Please make sure to read the |postgresql| documentation about the backup and res tore process: https://www.postgresql.org/docs/current/continuous-archiving.html
This is just a short outline of the tasks performed by the plugin. This is just a short outline of the tasks performed by the plugin.
#. Notify Postgres that we want to start backup the database files using the *SE LECT pg_start_backup()* statement #. Notify |postgresql| that we want to start backup the database files using the *SELECT pg_start_backup()* statement
#. Backup database files #. Backup database files
#. Notify Postgres when done with file backups using the *SELECT pg_stop_backup( #. Notify |postgresql| when done with file backups using the *SELECT pg_stop_bac
)* statement kup()* statement
#. Postgres will write *Write-Ahead-Logfiles* - WAL - into the WAL Archive. Thes #. |postgresql| will write *Write-Ahead-Logfiles* (WAL) into the WAL Archive dir
e transaction logs contain transactions done while the file backup proceeded ectory. These transaction logs contain transactions done while the file backup p
roceeded
#. Backup fresh created WAL files #. Backup fresh created WAL files
Incremental and Differential backups will only have to backup WAL files created since last reference backup. Incremental and Differential backups will only have to backup WAL files created since last reference backup.
The restore basically works like this: The restore basically works like this:
#. Restore all files to the original Postgres location #. Restore all files to the original |postgresql| location
#. Create a recovery.conf file (see below) #. Configure |postgresql| for the recovery (see below)
#. Start Postgres #. Start |postgresql|
#. Postgres will restore the latest possible consistent point in time. You can m #. |postgresql| will restore the latest possible consistent point in time. You c
anage to restore to any other point in in time available in the WAL files, pleas an manage to restore to any other point in in time available in the WAL files, p
e refer to the Postgres documentation for more details. lease refer to the |postgresql| documentation for more details.
Prerequisites Prerequisites
^^^^^^^^^^^^^ ^^^^^^^^^^^^^
As it is a Python plugin, it will also require to have the package **bareos-file This plugin is a Bareos Python plugin.
daemon-python-plugin** installed on the |fd|, where you run it. It requires |postgresql| >= 9 and the Python module **psycopg2** to be installed
.
Best use Python >= 3.
The plugin must be installed on the same host where the |postgresql| database ru
ns.
**You have to enable PostgreSQL WAL-Archiving** - the process and the plugin dep
end on it.
As a minimum this requires that you create an WAL archive directory
and matching settings in your |postgresql| configuration file **postgres.conf**.
In our examples we assume the WAL archive directory as :file:`/var/lib/pgsql/wal
_archive/`.
.. code-block:: cfg
:caption: postgres.conf
...
# wal_level default is replica
wal_level = replica
archive_mode = on
archive_command = 'test ! -f /var/lib/pgsql/wal_archive/%f && cp %p /var/lib/
pgsql/wal_archive/%f'
...
Please refer to the |postgresql| documentation for details.
The plugin requires the Python module psycopg2 to be installed in your python2 e nvironment. .. note::
**You have to enable WAL-Archiving** - the process and the plugin depend on it. While the PostgreSQL plugin backups only the required files from the WAL arch
ive directory,
old files are not removed automatically.
Installation Installation
^^^^^^^^^^^^ ^^^^^^^^^^^^
Make sure you have met the prerequisites, after that install the package **bareo s-filedaemon-postgres-python-plugin**. Make sure you have met the prerequisites, after that install the package **bareo s-filedaemon-postgres-python-plugin**.
The plugin must be installed on the same server where the |postgresql| database
runs.
Configuration Configuration
^^^^^^^^^^^^^ ^^^^^^^^^^^^^
Activate your plugin directory in the |fd| configuration. See :ref:`fdPlugins` f or more about plugins in general. Activate your plugin directory in the |fd| configuration. See :ref:`fdPlugins` f or more about plugins in general.
.. code-block:: bareosconfig .. code-block:: bareosconfig
:caption: bareos-fd.d/client/myself.conf :caption: bareos-fd.d/client/myself.conf
Client { Client {
... ...
Plugin Directory = /usr/lib64/bareos/plugins Plugin Directory = /usr/lib64/bareos/plugins
Plugin Names = "python" Plugin Names = "python3"
} }
Now include the plugin as command-plugin in the fileset resource and define a jo b using this fileset: Now include the plugin as command-plugin in the fileset resource and define a jo b using this fileset:
.. code-block:: bareosconfig .. code-block:: bareosconfig
:caption: bareos-dir.d/fileset/postgres.conf :caption: bareos-dir.d/fileset/postgres.conf
FileSet { FileSet {
Name = "postgres" Name = "postgres"
Include { Include {
skipping to change at line 1839 skipping to change at line 1863
signature = MD5 signature = MD5
} }
Plugin = "python" Plugin = "python"
":module_path=/usr/lib64/bareos/plugins" ":module_path=/usr/lib64/bareos/plugins"
":module_name=bareos-fd-postgres" ":module_name=bareos-fd-postgres"
":postgresDataDir=/var/lib/pgsql/data" ":postgresDataDir=/var/lib/pgsql/data"
":walArchive=/var/lib/pgsql/wal_archive/" ":walArchive=/var/lib/pgsql/wal_archive/"
} }
} }
You can append options to the plugin call as key=value pairs, separated by ’:’. The following options are available: You can append options to the plugin call as key=value pairs, separated by ``:`` . The following options are available:
- :strong:`postgresDataDir` the Postgres data directory. Default: :file:`/var/l postgresDataDir
ib/pgsql/data` the Postgres data directory. Default: :file:`/var/lib/pgsql/data`
- :strong:`walArchive` directory where Postgres archives the WAL files as defin walArchive
ed in your :file:`postgresql.conf` with the *archive_command* directive. This is directory where Postgres archives the WAL files as defined in your :file:`pos
a **mandatory** option, there is no default set. tgresql.conf` with the *archive_command* directive. This is a **mandatory** opti
on, there is no default set.
- :strong:`dbuser` with this user the plugin will try to connect to the databas dbuser
e. Default: *root* with this user the plugin will try to connect to the database. Default: *root
*
- :strong:`dbname` there needs to be a named database for the connection. Defau dbname
lt: *postgres* there needs to be a named database for the connection. Default: *postgres*
- :strong:`dbHost` useful, if socket is not in default location. Specify socket- dbHost
directory with a leading / here useful, if socket is not in default location. Specify socket-directory with a
leading / here
- :strong:`ignoreSubdirs` a list of comma separated directories below the *post ignoreSubdirs
gresDataDir*, that will not be backed up. Default: *pg_wal,pg_log,pg_xlog* a list of comma separated directories below the *postgresDataDir*, that will
not be backed up. Default: *pg_wal,pg_log,pg_xlog*
- :strong:`switchWal` If set to *true* (default), the plugin will let Postgres switchWal
write a new wal file, if the current Log Sequence Number (LSN) is greater than t If set to *true* (default), the plugin will let Postgres write a new wal file
he LSN from the previous job to make sure changes will go into the backup. , if the current Log Sequence Number (LSN) is greater than the LSN from the prev
ious job to make sure changes will go into the backup. Default: *true*
Restore Restore
^^^^^^^ ^^^^^^^
With the usual Bareos restore mechanism a file-hierarchy will be created on the restore client under the default restore location according to the options set: With the usual Bareos restore mechanism a file-hierarchy will be created on the restore client under the default restore location according to the options set:
- :file:`<restore prefix>/<postgresDataDir>/` - :file:`<restore prefix>/<postgresDataDir>/`
- :file:`<restore prefix>/<walArchive>/` - :file:`<restore prefix>/<walArchive>/`
You need to place a minimal :file:`recovery.conf` in your Postgres datadir, Exam This example describes how to restore to the latest possible consistent point in
ple: time. You can manage to restore to any other point in in time available in the
WAL files, please refer to the |postgresql| documentation for more details.
PostgreSQL >= 12
''''''''''''''''
Beginning with |postgresql| >= 12 the configuration must be done in your |postgr
esql| configuration file :file:`postgres.conf`:
.. code-block:: cfg
:caption: postgres.conf
...
restore_command = 'cp /var/lib/pgsql/wal_archive/%f %p'
...
Additionally a (empty) file named :file:`recovery.signal` must be created in you
r |postgresql| datadir.
PostgreSQL < 12
'''''''''''''''
For |postgresql| < 12 you need to place a minimal :file:`recovery.conf` in your
|postgresql| datadir.
Example:
.. code-block:: cfg .. code-block:: cfg
:caption: recovery.conf :caption: recovery.conf
restore_command = 'cp /var/lib/pgsql/wal_archive/%f %p' restore_command = 'cp /var/lib/pgsql/wal_archive/%f %p'
Where :file:`/var/lib/pgsql/wal_archive/` is the *walArchive* directory. Startin Where :file:`/var/lib/pgsql/wal_archive/` is the *walArchive* directory.
g the |postgresql| server shall now initiate the recovery process. Make sure tha
t the user *postgres* is allowed to rename the :file:`recovery.conf` file. You m Initiate the Recovery Process
ight have to disable or adapt your SELINUX configuration on some installations. '''''''''''''''''''''''''''''
Make sure that the user **postgres** is allowed to rename the recovery marker fi
le (:file:`recovery.signal` or :file:`recovery.conf`),
as the file will be renamed during the recovery process.
You might have to adapt your SELINUX configuration for this.
Starting the |postgresql| server shall now initiate the recovery process.
Troubleshooting Troubleshooting
^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^
If things don't work as expected, make sure that If things don't work as expected, make sure that
- the |fd| (FD) works in general, so that you can make simple file backups and r estores - the |fd| (FD) works in general, so that you can make simple file backups and r estores
- the Bareos FD Python plugins work in general, try one of - the Bareos FD Python plugins work in general, try one of
the shipped simple sample plugins the shipped simple sample plugins
- check your Postgres data directory for a file named backup_label. If it exists , another backup-process is already running. This file contains an entry like *L ABEL: SomeLabel*. If the backup was triggered by this plugin, the label will loo k like: *LABEL: Bareos.pgplugin.jobid.<jobid>*. - check your Postgres data directory for a file named backup_label. If it exists , another backup-process is already running. This file contains an entry like *L ABEL: SomeLabel*. If the backup was triggered by this plugin, the label will loo k like: *LABEL: Bareos.pgplugin.jobid.<jobid>*.
You may want to stop it using the *SELECT pg_stop_backup()* statement. You may want to stop it using the *SELECT pg_stop_backup()* statement.
- make sure your *dbuser* can connect to the database *dbname* and is allowed to - make sure your *dbuser* can connect to the database *dbname* and is allowed to
issue the following statements: issue the following statements matching your |postgresql| version:
.. code-block:: sql
SELECT current_setting('server_version_num') .. code-block:: sql
-- Postgres version >= 9
SELECT pg_start_backup()
SELECT pg_backup_start_time()"
SELECT pg_stop_backup()
-- Postgres version >=10:
SELECT pg_current_wal_lsn()
SELECT pg_switch_wal()
-- Postgres version 9 only:
SELECT pg_current_xlog_location()
SELECT pg_switch_xlog()
Support is available here: https://www.bareos.com SELECT current_setting('server_version_num');
-- Postgres version >= 9
SELECT pg_start_backup();
SELECT pg_backup_start_time();
SELECT pg_stop_backup();
-- Postgres version >=10:
SELECT pg_current_wal_lsn();
SELECT pg_switch_wal();
-- Postgres version 9 only:
SELECT pg_current_xlog_location();
SELECT pg_switch_xlog();
.. _sdPlugins: .. _sdPlugins:
Storage Daemon Plugins Storage Daemon Plugins
---------------------- ----------------------
.. _plugin-autoxflate-sd: .. _plugin-autoxflate-sd:
autoxflate-sd autoxflate-sd
~~~~~~~~~~~~~ ~~~~~~~~~~~~~
 End of changes. 30 change blocks. 
75 lines changed or deleted 141 lines changed or added

Home  |  About  |  Features  |  All  |  Newest  |  Dox  |  Diffs  |  RSS Feeds  |  Screenshots  |  Comments  |  Imprint  |  Privacy  |  HTTP(S)