"Fossies" - the Fresh Open Source Software Archive  

Source code changes of the file "docs/manual/secretsmanagement.rst" between
buildbot-3.0.2.tar.gz and buildbot-3.1.0.tar.gz

About: Buildbot is a continuous integration testing framework (Python-based). It supports also automation of complex build systems, application deployment, and management of sophisticated software-release processes.

secretsmanagement.rst  (buildbot-3.0.2):secretsmanagement.rst  (buildbot-3.1.0)
skipping to change at line 14 skipping to change at line 14
================= =================
Secret Management Secret Management
================= =================
Requirements Requirements
============ ============
Buildbot steps might need secrets to execute their actions. Buildbot steps might need secrets to execute their actions.
Secrets are used to execute commands or to create authenticated network connecti ons. Secrets are used to execute commands or to create authenticated network connecti ons.
Secrets may be a SSH key, a password, or a file content like a wgetrc file or a public SSH key. Secrets may be a SSH key, a password, or a file content like a wgetrc file or a public SSH key.
To preserve confidentiality, the secrets values must not be printed or logged in the twisted or steps logs. To preserve confidentiality, the secret values must not be printed or logged in the twisted or step logs.
Secrets must not be stored in the Buildbot configuration (master.cfg), as the so urce code is usually shared in SCM like git. Secrets must not be stored in the Buildbot configuration (master.cfg), as the so urce code is usually shared in SCM like git.
How to use Buildbot Secret Management How to use Buildbot Secret Management
===================================== =====================================
Secrets and providers Secrets and providers
--------------------- ---------------------
Buildbot implements several providers for secrets retrieval: Buildbot implements several providers for secrets retrieval:
- File system based: secrets are written in a file. - File system based: secrets are written in a file.
This is a simple solution for example when secrets are managed by config manag ement system like Ansible Vault. This is a simple solution for example when secrets are managed by a config man agement system like Ansible Vault.
- Third party backend based: secrets are stored by a specialized software. - Third party backend based: secrets are stored by a specialized software.
These solution are usually more secured. These solutions are usually more secure.
Secrets providers are configured if needed in the master configuration. Secrets providers are configured if needed in the master configuration.
Multiple providers can be configured at once. Multiple providers can be configured at once.
The secret manager is a Buildbot service. The secret manager is a Buildbot service.
The secret manager returns the specific provider results related to the provider s registered in the configuration. The secret manager returns the specific provider results related to the provider s registered in the configuration.
How to use secrets in Buildbot How to use secrets in Buildbot
------------------------------ ------------------------------
Secret can be used in Buildbot via the :class:`~IRenderable` mechanism. Secret can be used in Buildbot via the :class:`~IRenderable` mechanism.
Two :class:`~IRenderable` actually implement secrets. Two :class:`~IRenderable` actually implement secrets.
:ref:`Interpolate` can be used if you need to mix secrets and other interpolatio n in the same argument. :ref:`Interpolate` can be used if you need to mix secrets and other interpolatio n in the same argument.
:ref:`Interpolate` can be used if your secret is directly used as a component ar gument. :ref:`Secret` can be used if your secret is directly used as a component argumen t.
.. _Secret: .. _Secret:
Secret Secret
`````` ``````
:ref:`Secret` is a simple renderable which directly renders a secret. :ref:`Secret` is a simple renderable which directly renders a secret.
.. code-block:: python .. code-block:: python
Secret("secretName") Secret("secretName")
As argument to steps As argument to steps
```````````````````` ````````````````````
The following example shows a basic usage of secrets in Buildbot. The following example shows a basic usage of secrets in Buildbot.
.. code-block:: python .. code-block:: python
from buildbot.plugins import secrets, util from buildbot.plugins import secrets, util
# First we declare that the secrets are stored in a directory of the filesys tem # First we declare that the secrets are stored in a directory of the filesys tem
# each file contain one secret identified by the filename # each file contains one secret identified by the filename
c['secretsProviders'] = [secrets.SecretInAFile(dirname="/path/toSecretsFiles ")] c['secretsProviders'] = [secrets.SecretInAFile(dirname="/path/toSecretsFiles ")]
# then in a buildfactory: # then in a buildfactory:
# use a secret on a shell command via Interpolate # use a secret on a shell command via Interpolate
f1.addStep(ShellCommand( f1.addStep(ShellCommand(
util.Interpolate("wget -u user -p '%(secret:userpassword)s' '%(prop:urlt ofetch)s'"))) util.Interpolate("wget -u user -p '%(secret:userpassword)s' '%(prop:urlt ofetch)s'")))
# .. or non shell form: # .. or non shell form:
f1.addStep(ShellCommand(["wget", "-u", "user", "-p", f1.addStep(ShellCommand(["wget", "-u", "user", "-p",
util.Secret("userpassword"), util.Secret("userpassword"),
skipping to change at line 88 skipping to change at line 88
As argument to services As argument to services
``````````````````````` ```````````````````````
You can use secrets to configure services. You can use secrets to configure services.
All services arguments are not compatible with secrets. All services arguments are not compatible with secrets.
See their individual documentation for details. See their individual documentation for details.
.. code-block:: python .. code-block:: python
# First we declare that the secrets are stored in a directory of the filesys tem # First we declare that the secrets are stored in a directory of the filesys tem
# each file contain one secret identified by the filename # each file contains one secret identified by the filename
c['secretsProviders'] = [secrets.SecretInAFile(dirname="/path/toSecretsFiles ")] c['secretsProviders'] = [secrets.SecretInAFile(dirname="/path/toSecretsFiles ")]
# then for a reporter: # then for a reporter:
c['services'] = [GitHubStatusPush(token=util.Secret("githubToken"))] c['services'] = [GitHubStatusPush(token=util.Secret("githubToken"))]
Secrets storages Secrets storages
---------------- ----------------
.. _SecretInAFile: .. _SecretInAFile:
skipping to change at line 134 skipping to change at line 134
c['secretsProviders'] = [secrets.HashiCorpVaultSecretProvider( c['secretsProviders'] = [secrets.HashiCorpVaultSecretProvider(
vaultToken=open('VAULT_TOKEN').read().strip(), vaultToken=open('VAULT_TOKEN').read().strip(),
vaultServer="http://localhost:8200", vaultServer="http://localhost:8200",
secretsmount="secret", secretsmount="secret",
apiVersion=2 apiVersion=2
)] )]
Vault secures, stores, and tightly controls access to secrets. Vault secures, stores, and tightly controls access to secrets.
Vault presents a unified API to access multiple backends. Vault presents a unified API to access multiple backends.
At the moment Buildbot supports KV v1 and v2 backends via the apiVersion argumen t. At the moment, Buildbot supports KV v1 and v2 backends via the apiVersion argume nt.
Buildbot's Vault authentication/authorisation is via a token. Buildbot's Vault authentication/authorisation is via a token.
The "Initial Root Token", generated on Vault initialization, can be used but has ‘root’ authorization. The "Initial Root Token", generated on Vault initialization, can be used but has ‘root’ authorization.
Vault policies, and subsequent tokens assigned to them, provide for a more restr ictive approach. Vault policies, and subsequent tokens assigned to them, provide for a more restr ictive approach.
In the master configuration, the Vault provider is instantiated through the Buil dbot service manager as a secret provider with the Vault server address and the Vault token. In the master configuration, the Vault provider is instantiated through the Buil dbot service manager as a secret provider with the Vault server address and the Vault token.
The provider SecretInVault allows Buildbot to read secrets in Vault. The provider SecretInVault allows Buildbot to read secrets in Vault.
For more information about Vault please visit: _`Vault`: https://www.vaultprojec t.io/ For more information about Vault please visit: _`Vault`: https://www.vaultprojec t.io/
The secret identifiers that need to be passed to e.g. :ref:`Interpolate` accept one of the following The secret identifiers that need to be passed to, e.g. :ref:`Interpolate`, accep t one of the following
formats: formats:
- ``key``: The provider will fetch the secret with name ``key`` and return the value of ``value`` attribute stored therein. - ``key``: The provider will fetch the secret with name ``key`` and return the value of ``value`` attribute stored therein.
- ``key/attr``: The provider will fetch the secret with name ``key`` and return the value of ``attr`` attribute stored therein. - ``key/attr``: The provider will fetch the secret with name ``key`` and return the value of ``attr`` attribute stored therein.
Vault stores secrets in form of key value pairs. Vault stores secrets in form of key-value pairs.
- Simple keys - Simple keys
.. image:: _images/vault_simple_key.png .. image:: _images/vault_simple_key.png
The key value with key name ``keyname`` can be read like: The key value with key name ``keyname`` can be read like:
.. code-block:: python .. code-block:: python
text = Interpolate("your key equals %(secret:folder1/folder2/secretname/keyn ame)s") text = Interpolate("your key equals %(secret:folder1/folder2/secretname/keyn ame)s")
- Multipart keys - Multipart keys
.. image:: _images/vault_multipart_key.png .. image:: _images/vault_multipart_key.png
Each part of a multipart value can be read like Each part of a multipart value can be read like
.. code-block:: python .. code-block:: python
url = Interpolate("site url is %(secret: folder1/folde2/folde3/secretname/ur url = Interpolate("site url is %(secret:folder1/folde2/folde3/secretname/url
l)s") )s")
pass = Interpolate("your password is %(secret: folder1/folde2/folde3/secretn pass = Interpolate("your password is %(secret:folder1/folde2/folde3/secretna
ame/pass)s") me/pass)s")
cert = Interpolate("your cert is %(secret: folder1/folde2/folde3/secretname/ cert = Interpolate("your cert is %(secret:folder1/folde2/folde3/secretname/s
ssh-cert)s") sh-cert)s")
.. _SecretInPass: .. _SecretInPass:
SecretInPass SecretInPass
````````````` `````````````
.. code-block:: python .. code-block:: python
c['secretsProviders'] = [secrets.SecretInPass( c['secretsProviders'] = [secrets.SecretInPass(
gpgPassphrase="passphrase", gpgPassphrase="passphrase",
skipping to change at line 213 skipping to change at line 213
---------------------------------- ----------------------------------
To populate secrets in files during a build, 2 steps are used to create and dele te the files on the worker. To populate secrets in files during a build, 2 steps are used to create and dele te the files on the worker.
The files will be automatically deleted at the end of the build. The files will be automatically deleted at the end of the build.
.. code-block:: python .. code-block:: python
f = BuildFactory() f = BuildFactory()
with f.withSecrets(secrets_list): with f.withSecrets(secrets_list):
f.addStep(step_definition) f.addStep(step_definition)
or or
.. code-block:: python .. code-block:: python
f = BuildFactory() f = BuildFactory()
f.addSteps([list_of_step_definitions], withSecrets=[secrets_list]) f.addSteps([list_of_step_definitions], withSecrets=[secrets_list])
In both cases the secrets_list is a list of tuple (secret path, secret value). In both cases the secrets_list is a list of (secret path, secret value) tuples.
.. code-block:: python .. code-block:: python
secrets_list = [('/first/path', Interpolate('write something and %(secre t:somethingmore)s')), secrets_list = [('/first/path', Interpolate('write something and %(secre t:somethingmore)s')),
('/second/path', Interpolate('%(secret:othersecret)s')] ('/second/path', Interpolate('%(secret:othersecret)s')]
The Interpolate class is used to render the value during the build execution. The Interpolate class is used to render the value during the build execution.
How to configure a Vault instance How to configure a Vault instance
--------------------------------- ---------------------------------
skipping to change at line 270 skipping to change at line 271
Then, export the environment variable VAULT_ADDR needed to init Vault. Then, export the environment variable VAULT_ADDR needed to init Vault.
.. code-block:: shell .. code-block:: shell
export VAULT_ADDR='vault.server.adress' export VAULT_ADDR='vault.server.adress'
Writing secrets Writing secrets
``````````````` ```````````````
By default the official docker instance of Vault is initialized with a mount pat h of 'secret', a KV v1 secret engine, and a second KV engine (v2) at 'secret/dat a'. By default the official docker instance of Vault is initialized with a mount pat h of 'secret', a KV v1 secret engine, and a second KV engine (v2) at 'secret/dat a'.
Currently Buildbot is "hard wired" to expect KV v2 engines to reside within this "data" sub path. Currently, Buildbot is "hard wired" to expect KV v2 engines to reside within thi s "data" sub path.
Provision is made to set a top level path via the "secretsmount" argument: defau lts to "secret". Provision is made to set a top level path via the "secretsmount" argument: defau lts to "secret".
To add a new secret: To add a new secret:
.. code-block:: shell .. code-block:: shell
vault kv put secret/new_secret_key value=new_secret_value vault kv put secret/new_secret_key value=new_secret_value
 End of changes. 13 change blocks. 
17 lines changed or deleted 18 lines changed or added

Home  |  About  |  Features  |  All  |  Newest  |  Dox  |  Diffs  |  RSS Feeds  |  Screenshots  |  Comments  |  Imprint  |  Privacy  |  HTTP(S)