"Fossies" - the Fresh Open Source Software Archive  

Source code changes of the file "docs/manual/concepts.rst" between
buildbot-3.0.2.tar.gz and buildbot-3.1.0.tar.gz

About: Buildbot is a continuous integration testing framework (Python-based). It supports also automation of complex build systems, application deployment, and management of sophisticated software-release processes.

concepts.rst  (buildbot-3.0.2):concepts.rst  (buildbot-3.1.0)
.. _Concepts: .. _Concepts:
Concepts Concepts
======== ========
This chapter defines some of the basic concepts that the Buildbot uses. This chapter defines some of the basic concepts that Buildbot uses.
You'll need to understand how the Buildbot sees the world to configure it proper You'll need to understand how Buildbot sees the world to configure it properly.
ly.
.. index: repository .. index: repository
.. index: codebase .. index: codebase
.. index: project .. index: project
.. index: revision .. index: revision
.. index: branch .. index: branch
.. index: source stamp .. index: source stamp
.. _Source-Stamps: .. _Source-Stamps:
Source identification Source identification
--------------------- ---------------------
The following concepts are used within Buildbot to describe source code that is being built: The following concepts are used within Buildbot to describe source code that is being built:
Repository Repository
A repository is a location where files tracked by a version control system r eside. A repository is a location where files tracked by a version control system r eside.
Usually it is identified by a URL or a location on a disk. Usually, it is identified by a URL or a location on a disk.
It contains a subset of history of a codebase. It contains a subset of the history of a codebase.
Codebase Codebase
A codebase is a collection of related files and their history tracked as a u nit by version control systems. A codebase is a collection of related files and their history tracked as a u nit by version control systems.
The files and their history are stored in one or more repositories. The files and their history are stored in one or more repositories.
For example, the primary repository for the Buildbot codebase is at ``https: //github.com/buildbot/buildbot/``. For example, the primary repository for the Buildbot codebase is at ``https: //github.com/buildbot/buildbot/``.
There are also more than a thousand forks of Buildbot. There are also more than a thousand forks of Buildbot.
These repositories, while storing potentially very old versions of Buildbot code, still contain the same codebase. These repositories, while storing potentially very old versions of Buildbot code, still contain the same codebase.
Project Project
A project is a set of one or more codebases that together may be built and p roduce some end artifact. A project is a set of one or more codebases that together may be built and p roduce some end artifact.
For example, an application may be comprised of two codebases - one for the code and one for the test data the latter of which occupies a lot of space. For example, an application may be comprised of two codebases - one for the code and one for the test data, the latter of which occupies a lot of space.
Building and testing such an application requires acquiring code from both c odebases. Building and testing such an application requires acquiring code from both c odebases.
Revision: Revision:
A revision is an textual identifier used by most version control systems to uniquely specify a particular version of the source code is a particular codebas e. A revision is an textual identifier used by most version control systems to uniquely specify a particular version of the source code in a particular codebas e.
Source stamp: Source stamp:
A source stamp is a collection of information needed to identify a particula r version of code on a certain codebase. A source stamp is a collection of information needed to identify a particula r version of code on a certain codebase.
In most version control systems source stamp only stores a revision. In most version control systems, source stamps only store a revision.
On certain version control systems a branch is also required. On other version control systems, a branch is also required.
Source stamp set: Source stamp set:
A source stamp set is a set of source stamps to identify a particular versio n of code on a certain project. A source stamp set is a set of source stamps to identify a particular versio n of code on a certain project.
Like a project is a collection of codebases, a source stamp set is a collect ion of source stamps, one for each codebase within a project. Like a project is a collection of codebases, a source stamp set is a collect ion of source stamps, one for each codebase within a project.
In order to build a project Buildbot only needs to know a source stamp set corre sponding to that project. In order to build a project, Buildbot only needs to know a source stamp set corr esponding to that project.
This source stamp set has a source stamp for each codebase comprising the projec t. This source stamp set has a source stamp for each codebase comprising the projec t.
In turn, each source stamp has enough information to identify a particular versi on of the code within the codebase. In turn, each source stamp has enough information to identify a particular versi on of the code within the codebase.
.. image:: _images/changes.* .. image:: _images/changes.*
:alt: Source Stamp Sets :alt: Source Stamp Sets
.. _Concepts-Change-Source: .. _Concepts-Change-Source:
Change sources Change sources
-------------- --------------
Change sources are user-configurable components that interact with external vers ion control systems and retrieve new code. Change sources are user-configurable components that interact with external vers ion control systems and retrieve new code.
Internally new code is represented as :ref:`Changes <Concept-Change>` which roug hly correspond to single commit or changeset. Internally, new code is represented as :ref:`Changes <Concept-Change>` which rou ghly correspond to a single commit or changeset.
The changes are sent to the schedulers which then decide whether new builds shou ld be created for these new code changes. The changes are sent to the schedulers which then decide whether new builds shou ld be created for these new code changes.
The design of Buildbot requires the workers to have their own copies of the sour ce code, thus change sources is an optional component as long as there are no sc hedulers that create new builds based on new code commit events. The design of Buildbot requires the workers to have their own copies of the sour ce code, thus change sources is an optional component as long as there are no sc hedulers that create new builds based on new code commit events.
.. index: change .. index: change
.. _Concept-Change: .. _Concept-Change:
Changes Changes
------- -------
A :ref:`Change<Change-Attrs>` is an abstract way Buildbot uses to represent a si A :ref:`Change<Change-Attrs>` is an abstract way Buildbot uses to represent a si
ngle change to the source files performed by a developer. ngle change to the source files, performed by a developer.
In version control systems that support the notion of atomic check-ins a change In version control systems that support the notion of atomic check-ins, a change
represents a changeset or commit. represents a changeset or commit.
Changes are used for the :ref:`Change sources<Concepts-Change-Source>` to commun icate with :ref:`Schedulers <Concepts-Scheduler>`. Changes are used for the :ref:`Change sources<Concepts-Change-Source>` to commun icate with :ref:`Schedulers <Concepts-Scheduler>`.
A :class:`Change` comprises the following information: A :class:`Change` comprises the following information:
- the developer that is responsible for the change - the developer who is responsible for the change
- the list of files that the change added, removed or modified - the list of files that the change added, removed or modified
- the message of the commit - the message of the commit
- the repository, the codebase and the project that the change corresponds to - the repository, the codebase and the project that the change corresponds to
- the revision and the branch of the commit - the revision and the branch of the commit
.. _Concepts-Scheduler: .. _Concepts-Scheduler:
Schedulers Schedulers
---------- ----------
A scheduler is a component that decides when to start a build. A scheduler is a component that decides when to start a build.
The decision could be based on time, on new code being committed or on similar e vents. The decision could be based on time, on new code being committed or on similar e vents.
Schedulers are responsible for creating :ref:`Build Requests<Concepts-Build-Requ est>` which identify a request to start a build on a specific versions of the so urce code. Schedulers are responsible for creating :ref:`Build Requests<Concepts-Build-Requ est>` which identify a request to start a build on a specific version of the sou rce code.
Each Buildmaster has a set of scheduler objects, each of which gets a copy of ev ery incoming :class:`Change`. Each Buildmaster has a set of scheduler objects, each of which gets a copy of ev ery incoming :class:`Change`.
The Schedulers are responsible for deciding when :class:`Build`\s should be run. The Schedulers are responsible for deciding when :class:`Build`\s should be run.
Some Buildbot installations might have a single scheduler, while others may have several, each for a different purpose. Some Buildbot installations might have a single scheduler, while others may have several, each for a different purpose.
.. _Concepts-Build-Request: .. _Concepts-Build-Request:
BuildRequests BuildRequests
------------- -------------
A :class:`BuildRequest` is a request to start a specific build. A :class:`BuildRequest` is a request to start a specific build.
A :class:`BuildRequest` consists of the following information: A :class:`BuildRequest` consists of the following information:
- the name of the :class:`Builder` (see below) that will start the build. - the name of the :class:`Builder` (see below) that will perform the build.
- the set of :class:`SourceStamp`\s (see above) that specify the version of the source tree to build and/or test. - the set of :class:`SourceStamp`\s (see above) that specify the version of the source tree to build and/or test.
Two build requests representing the same version of the source code and the same builder may be merged. Two build requests representing the same version of the source code and the same builder may be merged.
The user may configure additional restrictions for determining mergeability of b uild requests. The user may configure additional restrictions for determining mergeability of b uild requests.
.. _Concepts-Builder: .. _Concepts-Builder:
.. _Concepts-Build-Factories: .. _Concepts-Build-Factories:
skipping to change at line 157 skipping to change at line 157
The directory on the master is used for keeping status information. The directory on the master is used for keeping status information.
The directories on the workers are used as a location where the actual checkout, compilation and testing steps happen. The directories on the workers are used as a location where the actual checkout, compilation and testing steps happen.
.. _Concepts-Build: .. _Concepts-Build:
.. _Concepts-Step: .. _Concepts-Step:
Builds Builds
------ ------
A :class:`Build` represents a single compile or test run of a particular version of the source code. A :class:`Build` represents a single compile or test run of a particular version of a source code.
A build is comprised of a series of steps. A build is comprised of a series of steps.
The steps may be arbitrary. For example, for compiled software a build generally consists of the checkout, configure, make, and make check sequence. The steps may be arbitrary. For example, for compiled software a build generally consists of the checkout, configure, make, and make check sequence.
For interpreted projects like Python modules, a build is generally a checkout fo llowed by an invocation of the bundled test suite. For interpreted projects like Python modules, a build is generally a checkout fo llowed by an invocation of the bundled test suite.
Builds are created by instances of :class:`Builder` (see above). Builds are created by instances of :class:`Builder` (see above).
.. _Concepts-BuildSet: .. _Concepts-BuildSet:
BuildSets BuildSets
--------- ---------
skipping to change at line 189 skipping to change at line 189
A single physical machine must run at least one :class:`Worker` in order for Bui ldbot to be able to utilize it for running builds. A single physical machine must run at least one :class:`Worker` in order for Bui ldbot to be able to utilize it for running builds.
Multiple :class:`Worker`\s may run on a single machine to provide different envi ronments that can reuse the same hardware by means of containers or virtual mach ines. Multiple :class:`Worker`\s may run on a single machine to provide different envi ronments that can reuse the same hardware by means of containers or virtual mach ines.
Each builder is associated with one or more :class:`Worker`\s. Each builder is associated with one or more :class:`Worker`\s.
For example, a builder which is used to perform macOS builds (as opposed to Linu x or Windows builds) should naturally be associated with a Mac worker. For example, a builder which is used to perform macOS builds (as opposed to Linu x or Windows builds) should naturally be associated with a Mac worker.
If multiple workers are available for any given builder, you will have some meas ure of redundancy: in case one worker goes offline, the others can still keep th e :class:`Builder` working. If multiple workers are available for any given builder, you will have some meas ure of redundancy: in case one worker goes offline, the others can still keep th e :class:`Builder` working.
In addition, multiple workers will allow multiple simultaneous builds for the sa me :class:`Builder`, which might be useful if you have a lot of forced or ``try` ` builds taking place. In addition, multiple workers will allow multiple simultaneous builds for the sa me :class:`Builder`, which might be useful if you have a lot of forced or ``try` ` builds taking place.
Ideally, each :class:`Worker` that is configured for a builder should be identic al. Ideally, each :class:`Worker` that is configured for a builder should be identic al.
Otherwise build or test failures will be dependent on which worker the build is ran and this will complicate investigation of failures. Otherwise build or test failures will be dependent on which worker the build is run and this will complicate investigations of failures.
.. _Concepts-Users: .. _Concepts-Users:
Users Users
----- -----
Buildbot has a somewhat limited awareness of *users*. Buildbot has a somewhat limited awareness of *users*.
It assumes the world consists of a set of developers, each of whom can be descri bed by a couple of simple attributes. It assumes the world consists of a set of developers, each of whom can be descri bed by a couple of simple attributes.
These developers make changes to the source code, causing builds which may succe ed or fail. These developers make changes to the source code, causing builds which may succe ed or fail.
skipping to change at line 224 skipping to change at line 224
User Objects User Objects
~~~~~~~~~~~~ ~~~~~~~~~~~~
User Objects allow Buildbot to better manage users throughout its various intera ctions with users (see :ref:`Change-Sources` and :ref:`Reporters`). User Objects allow Buildbot to better manage users throughout its various intera ctions with users (see :ref:`Change-Sources` and :ref:`Reporters`).
The User Objects are stored in the Buildbot database and correlate the various a ttributes that a user might have: irc, Git, etc. The User Objects are stored in the Buildbot database and correlate the various a ttributes that a user might have: irc, Git, etc.
Changes Changes
+++++++ +++++++
Incoming Changes all have a ``who`` attribute attached to them that specifies wh ich developer is responsible for that Change. Incoming Changes all have a ``who`` attribute attached to them that specifies wh ich developer is responsible for that Change.
When a Change is first rendered, the ``who`` attribute is parsed and added to th e database if it doesn't exist or checked against an existing user. When a Change is first rendered, the ``who`` attribute is parsed and added to th e database, if it doesn't exist, or checked against an existing user.
The ``who`` attribute is formatted in different ways depending on the version co ntrol system that the Change came from. The ``who`` attribute is formatted in different ways depending on the version co ntrol system that the Change came from.
``git`` ``git``
``who`` attributes take the form ``Full Name <Email>``. ``who`` attributes take the form ``Full Name <Email>``.
``svn`` ``svn``
``who`` attributes are of the form ``Username``. ``who`` attributes are of the form ``Username``.
``hg`` ``hg``
``who`` attributes are free-form strings, but usually adhere to similar conv entions as ``git`` attributes (``Full Name <Email>``). ``who`` attributes are free-form strings, but usually adhere to similar conv entions as ``git`` attributes (``Full Name <Email>``).
skipping to change at line 261 skipping to change at line 261
++++ ++++
Correlating the various bits and pieces that Buildbot views as users also means that one attribute of a user can be translated into another. Correlating the various bits and pieces that Buildbot views as users also means that one attribute of a user can be translated into another.
This provides a more complete view of users throughout Buildbot. This provides a more complete view of users throughout Buildbot.
One such use is being able to find email addresses based on a set of Builds to n otify users through the ``MailNotifier``. One such use is being able to find email addresses based on a set of Builds to n otify users through the ``MailNotifier``.
This process is explained more clearly in :ref:`Email-Addresses`. This process is explained more clearly in :ref:`Email-Addresses`.
Another way to utilize `User Objects` is through `UsersAuth` for web authenticat ion. Another way to utilize `User Objects` is through `UsersAuth` for web authenticat ion.
To use `UsersAuth`, you need to set a `bb_username` and `bb_password` via the `` buildbot user`` command line tool to check against. To use `UsersAuth`, you need to set a `bb_username` and `bb_password` via the `` buildbot user`` command line tool to check against.
The password will be encrypted before storing in the database along with other u ser attributes. The password will be encrypted before it gets stored in the database along with other user attributes.
.. _Doing-Things-With-Users: .. _Doing-Things-With-Users:
Doing Things With Users Doing Things With Users
~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~
Each change has a single user who is responsible for it. Each change has a single user who is responsible for it.
Most builds have a set of changes: the build generally represents the first time these changes have been built and tested by the Buildbot. Most builds have a set of changes: the build generally represents the first time these changes have been built and tested by the Buildbot.
The build has a *blamelist* that is the union of the users responsible for all t The build has a *blamelist* that is the union of the users responsible for all o
he build's changes. f the build's changes.
If the build was created by a :ref:`Try-Schedulers` this list will include the s If the build was created by a :ref:`Try-Schedulers` this list will include the s
ubmitter of the try job, if known. ubmitter of the try job if known.
The build provides a list of users who are interested in the build -- the *inter ested users*. The build provides a list of users who are interested in the build -- the *inter ested users*.
Usually this is equal to the blamelist, but may also be expanded, e.g., to inclu de the current build sherrif or a module's maintainer. Usually this is equal to the blamelist, but may also be expanded, e.g., to inclu de the current build sherrif or a module's maintainer.
If desired, the buildbot can notify the interested users until the problem is re solved. If desired, buildbot can notify the interested users until the problem is resolv ed.
.. _Email-Addresses: .. _Email-Addresses:
Email Addresses Email Addresses
~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~
The :bb:reporter:`MailNotifier` is a status target which can send email about th The :bb:reporter:`MailNotifier` is a status target which can send emails about t
e results of each build. he results of each build.
It accepts a static list of email addresses to which each message should be deli It accepts a static list of email addresses to which each message should be deli
vered, but it can also be configured to send mail to the :class:`Build`\'s Inter vered, but it can also be configured to send emails to a :class:`Build`\'s Inter
ested Users. ested Users.
To do this, it needs a way to convert User names into email addresses. To do this, it needs a way to convert User names into email addresses.
For many VC systems, the User Name is actually an account name on the system whi For many VCSs, the User name is actually an account name on the system which hos
ch hosts the repository. ts the repository.
As such, turning the name into an email address is a simple matter of appending As such, turning the name into an email address is simply a matter of appending
``@repositoryhost.com``. ``@repositoryhost.com``.
Some projects use other kinds of mappings (for example the preferred email addre Some projects use other kinds of mappings (for example the preferred email addre
ss may be at ``project.org`` despite the repository host being named ``cvs.proje ss may be at ``project.org``, despite the repository host being named ``cvs.proj
ct.org``), and some VC systems have full separation between the concept of a use ect.org``), and some VCSs have full separation between the concept of a user and
r and that of an account on the repository host (like Perforce). that of an account on the repository host (like Perforce).
Some systems (like Git) put a full contact email address in every change. Some systems (like Git) put a full contact email address in every change.
To convert these names to addresses, the :class:`MailNotifier` uses an :class:`E mailLookup` object. To convert these names to addresses, the :class:`MailNotifier` uses an :class:`E mailLookup` object.
This provides a :meth:`getAddress` method which accepts a name and (eventually) returns an address. This provides a :meth:`getAddress` method which accepts a name and (eventually) returns an address.
The default :class:`MailNotifier` module provides an :class:`EmailLookup` which simply appends a static string, configurable when the notifier is created. The default :class:`MailNotifier` module provides an :class:`EmailLookup` which simply appends a static string, configurable when the notifier is created.
To create more complex behaviors (perhaps using an LDAP lookup, or using ``finge r`` on a central host to determine a preferred address for the developer), provi de a different object as the ``lookup`` argument. To create more complex behaviors (perhaps using an LDAP lookup, or using ``finge r`` on a central host to determine a preferred address for the developer), provi de a different object as the ``lookup`` argument.
If an EmailLookup object isn't given to the MailNotifier, the MailNotifier will try to find emails through :ref:`User-Objects`. If an EmailLookup object isn't given to the MailNotifier, the MailNotifier will try to find emails through :ref:`User-Objects`.
This will work the same as if an EmailLookup object was used if every user in th e Build's Interested Users list has an email in the database for them. If every user in the Build's Interested Users list has an email in the database for them, this will work the same as if an EmailLookup object was used.
If a user whose change led to a Build doesn't have an email attribute, that user will not receive an email. If a user whose change led to a Build doesn't have an email attribute, that user will not receive an email.
If ``extraRecipients`` is given, those users are still sent mail when the EmailL ookup object is not specified. If ``extraRecipients`` is given, those users still get an email when the EmailLo okup object is not specified.
In the future, when the Problem mechanism has been set up, the Buildbot will nee d to send mail to arbitrary Users. In the future, when the Problem mechanism has been set up, Buildbot will need to send emails to arbitrary Users.
It will do this by locating a :class:`MailNotifier`\-like object among all the b uildmaster's status targets, and asking it to send messages to various Users. It will do this by locating a :class:`MailNotifier`\-like object among all the b uildmaster's status targets, and asking it to send messages to various Users.
This means the User-to-address mapping only has to be set up once, in your :clas s:`MailNotifier`, and every email message the buildbot emits will take advantage of it. This means the User-to-address mapping only has to be set up once, in your :clas s:`MailNotifier`, and every email message buildbot emits will take advantage of it.
.. _IRC-Nicknames: .. _IRC-Nicknames:
IRC Nicknames IRC Nicknames
~~~~~~~~~~~~~ ~~~~~~~~~~~~~
Like :class:`MailNotifier`, the :class:`buildbot.reporters.irc.IRC` class provid es a status target which can announce the results of each build. Like :class:`MailNotifier`, the :class:`buildbot.reporters.irc.IRC` class provid es a status target which can announce the results of each build.
It also provides an interactive interface by responding to online queries posted in the channel or sent as private messages. It also provides an interactive interface by responding to online queries posted in the channel or sent as private messages.
In the future, the buildbot can be configured to map User names to IRC nicknames , to watch for the recent presence of these nicknames, and to deliver build stat us messages to the interested parties. In the future, buildbot can be configured to map User names to IRC nicknames, to watch for the recent presence of these nicknames, and to deliver build status m essages to the interested parties.
Like :class:`MailNotifier` does for email addresses, the :class:`IRC` object wil l have an :class:`IRCLookup` which is responsible for nicknames. Like :class:`MailNotifier` does for email addresses, the :class:`IRC` object wil l have an :class:`IRCLookup` which is responsible for nicknames.
The mapping can be set up statically, or it can be updated by online users thems elves (by claiming a username with some kind of ``buildbot: i am user warner`` c ommands). The mapping can be set up statically, or it can be updated by online users thems elves (by claiming a username with some kind of ``buildbot: i am user warner`` c ommands).
Once the mapping is established, the rest of the buildbot can ask the :class:`IR C` object to send messages to various users. Once the mapping is established, buildbot can then ask the :class:`IRC` object t o send messages to various users.
It can report on the likelihood that the user saw the given message (based upon how long the user has been inactive on the channel), which might prompt the Prob lem Hassler logic to send them an email message instead. It can report on the likelihood that the user saw the given message (based upon how long the user has been inactive on the channel), which might prompt the Prob lem Hassler logic to send them an email message instead.
These operations and authentication of commands issued by particular nicknames w ill be implemented in :ref:`User-Objects`. These operations and authentication of commands issued by particular nicknames w ill be implemented in :ref:`User-Objects`.
.. index:: Properties .. index:: Properties
.. _Build-Properties: .. _Build-Properties:
Build Properties Build Properties
---------------- ----------------
Each build has a set of *Build Properties*, which can be used by its build steps to modify their actions. Each build has a set of *Build Properties*, which can be used by its build steps to modify their actions.
The properties are represented as a set of key-value pairs. The properties are represented as a set of key-value pairs.
Effectively, a single property is a variable that, once set, can be used by subs equent steps in a build to modify their behaviour. Effectively, a single property is a variable that, once set, can be used by subs equent steps in a build to modify their behaviour.
The value of a property can be a number, a string, a list or a dictionary. The value of a property can be a number, a string, a list or a dictionary.
Lists and dictionaries can contain other lists or dictionaries. Lists and dictionaries can contain other lists or dictionaries.
Thus, the value of a property could be arbitrarily complex structure. Thus, the value of a property could be any arbitrarily complex structure.
Properties work pretty much like variables, so they can be used to implement all manner of functionality. Properties work pretty much like variables, so they can be used to implement all manner of functionality.
The following are several examples: The following are a couple of examples:
- By default, the name of the worker that runs the build is set to the ``worker name`` property. - By default, the name of the worker that runs the build is set to the ``worker name`` property.
If there are multiple different workers and the actions of the build depend o n the exact worker, some users may decide that it's more convenient to vary the actions depending on the ``workername`` property instead of creating separate bu ilders for each worker. If there are multiple different workers and the actions of the build depend o n the exact worker, some users may decide that it's more convenient to vary the actions depending on the ``workername`` property instead of creating separate bu ilders for each worker.
- In most cases the build does not know the exact code revision that will be te sted until it checks out the code. - In most cases, the build does not know the exact code revision that will be t ested until it checks out the code.
This information is only known after a :ref:`source step <Build-Steps>` runs. This information is only known after a :ref:`source step <Build-Steps>` runs.
To give this information to the subsequent steps, the source step records the checked out revision into the ``got_revision`` property. To give this information to the subsequent steps, the source step records the checked out revision into the ``got_revision`` property.
 End of changes. 28 change blocks. 
49 lines changed or deleted 48 lines changed or added

Home  |  About  |  Features  |  All  |  Newest  |  Dox  |  Diffs  |  RSS Feeds  |  Screenshots  |  Comments  |  Imprint  |  Privacy  |  HTTP(S)