"Fossies" - the Fresh Open Source Software Archive  

Source code changes of the file "docs/tutorial/fiveminutes.rst" between
buildbot-3.0.2.tar.gz and buildbot-3.1.0.tar.gz

About: Buildbot is a continuous integration testing framework (Python-based). It supports also automation of complex build systems, application deployment, and management of sophisticated software-release processes.

fiveminutes.rst  (buildbot-3.0.2):fiveminutes.rst  (buildbot-3.1.0)
.. _fiveminutes: .. _fiveminutes:
=================================================== ===================================================
Buildbot in 5 minutes - a user-contributed tutorial Buildbot in 5 minutes - a user-contributed tutorial
=================================================== ===================================================
(Ok, maybe 10.) (Ok, maybe 10.)
Buildbot is really an excellent piece of software, however it can be a bit confu sing for a newcomer (like me when I first started looking at it). Buildbot is really an excellent piece of software, however it can be a bit confu sing for a newcomer (like me when I first started looking at it).
Typically, at first sight it looks like a bunch of complicated concepts that mak e no sense and whose relationships with each other are unclear. Typically, at first sight, it looks like a bunch of complicated concepts that ma ke no sense and whose relationships with each other are unclear.
After some time and some reread, it all slowly starts to be more and more meanin gful, until you finally say "oh!" and things start to make sense. After some time and some reread, it all slowly starts to be more and more meanin gful, until you finally say "oh!" and things start to make sense.
Once you get there, you realize that the documentation is great, but only if you already know what it's about. Once you get there, you realize that the documentation is great, but only if you already know what it's about.
This is what happened to me, at least. This is what happened to me, at least.
Here I'm going to (try to) explain things in a way that would have helped me mor Here, I'm going to (try to) explain things in a way that would have helped me mo
e as a newcomer. re as a newcomer.
The approach I'm taking is more or less the reverse of that used by the document The approach I'm taking is more or less the reverse of that used by the document
ation, that is, I'm going to start from the components that do the actual work ( ation. That is, I'm going to start from the components that do the actual work (
the builders) and go up the chain from there up to change sources. the builders) and go up the chain to the change sources.
I hope purists will forgive this unorthodoxy. I hope purists will forgive this unorthodoxy.
Here I'm trying to clarify the concepts only, and will not go into the details o f each object or property; the documentation explains those quite well. Here I'm trying to only clarify the concepts and not go into the details of each object or property; the documentation explains those quite well.
Installation Installation
------------ ------------
I won't cover the installation; both Buildbot master and worker are available as packages for the major distributions, and in any case the instructions in the o fficial documentation are fine. I won't cover the installation; both Buildbot master and worker are available as packages for the major distributions, and in any case the instructions in the o fficial documentation are fine.
This document will refer to Buildbot 0.8.5 which was current at the time of writ This document will refer to Buildbot 0.8.5 which was current at the time of writ
ing, but hopefully the concepts are not too different in other versions. ing, but hopefully the concepts are not too different in future versions.
All the code shown is of course python code, and has to be included in the maste All the code shown is of course python code, and has to be included in the maste
r.cfg master configuration file. r.cfg configuration file.
We won't cover the basic things such as how to define the workers, project names , or other administrative information that is contained in that file; for that, again the official documentation is fine. We won't cover basic things such as how to define the workers, project names, or other administrative information that is contained in that file; for that, agai n the official documentation is fine.
Builders: the workhorses Builders: the workhorses
------------------------ ------------------------
Since Buildbot is a tool whose goal is the automation of software builds, it mak es sense to me to start from where we tell Buildbot how to build our software: t he `builder` (or builders, since there can be more than one). Since Buildbot is a tool whose goal is the automation of software builds, it mak es sense to me to start from where we tell Buildbot how to build our software: t he `builder` (or builders, since there can be more than one).
Simply put, a builder is an element that is in charge of performing some action or sequence of actions, normally something related to building software (for exa mple, checking out the source, or ``make all``), but it can also run arbitrary c ommands. Simply put, a builder is an element that is in charge of performing some action or sequence of actions, normally something related to building software (for exa mple, checking out the source, or ``make all``), but it can also run arbitrary c ommands.
A builder is configured with a list of workers that it can use to carry out its task. A builder is configured with a list of workers that it can use to carry out its task.
The other fundamental piece of information that a builder needs is, of course, t he list of things it has to do (which will normally run on the chosen worker). The other fundamental piece of information that a builder needs is, of course, t he list of things it has to do (which will normally run on the chosen worker).
In Buildbot, this list of things is represented as a ``BuildFactory`` object, wh ich is essentially a sequence of steps, each one defining a certain operation or command. In Buildbot, this list of things is represented as a ``BuildFactory`` object, wh ich is essentially a sequence of steps, each one defining a certain operation or command.
Enough talk, let's see an example. Enough talk, let's see an example.
For this example, we are going to assume that our super software project can be built using a simple ``make all``, and there is another target ``make packages`` that creates rpm, deb and tgz packages of the binaries. For this example, we are going to assume that our super software project can be built using a simple ``make all``, and there is another target ``make packages`` that creates rpm, deb and tgz packages of the binaries.
In the real world things are usually more complex (for example there may be a `` configure`` step, or multiple targets), but the concepts are the same; it will j ust be a matter of adding more steps to a builder, or creating multiple builders , although sometimes the resulting builders can be quite complex. In the real world things are usually more complex (for example there may be a `` configure`` step, or multiple targets), but the concepts are the same; it will j ust be a matter of adding more steps to a builder, or creating multiple builders , although sometimes the resulting builders can be quite complex.
So to perform a manual build of our project we would type this from the command line (assuming we are at the root of the local copy of the repository): So to perform a manual build of our project, we would type the following on the command line (assuming we are at the root of the local copy of the repository):
.. code-block:: bash .. code-block:: bash
$ make clean # clean remnants of previous builds $ make clean # clean remnants of previous builds
... ...
$ svn update $ svn update
... ...
$ make all $ make all
... ...
$ make packages $ make packages
skipping to change at line 96 skipping to change at line 96
haltOnFailure=True, haltOnFailure=True,
description="make all") description="make all")
# step 4: make packages # step 4: make packages
makepackages = steps.ShellCommand(name="make packages", makepackages = steps.ShellCommand(name="make packages",
command=["make", "packages"], command=["make", "packages"],
haltOnFailure=True, haltOnFailure=True,
description="make packages") description="make packages")
# step 5: upload packages to central server. This needs passwordless ssh # step 5: upload packages to central server. This needs passwordless ssh
# from the worker to the server (set it up in advance as part of worker setu p) # from the worker to the server (set it up in advance as part of the worker setup)
uploadpackages = steps.ShellCommand( uploadpackages = steps.ShellCommand(
name="upload packages", name="upload packages",
description="upload packages", description="upload packages",
command="scp packages/*.rpm packages/*.deb packages/*.tgz someuser@someh ost:/repository", command="scp packages/*.rpm packages/*.deb packages/*.tgz someuser@someh ost:/repository",
haltOnFailure=True) haltOnFailure=True)
# create the build factory and add the steps to it # create the build factory and add the steps to it
f_simplebuild = util.BuildFactory() f_simplebuild = util.BuildFactory()
f_simplebuild.addStep(makeclean) f_simplebuild.addStep(makeclean)
f_simplebuild.addStep(checkout) f_simplebuild.addStep(checkout)
f_simplebuild.addStep(makeall) f_simplebuild.addStep(makeall)
f_simplebuild.addStep(makepackages) f_simplebuild.addStep(makepackages)
f_simplebuild.addStep(uploadpackages) f_simplebuild.addStep(uploadpackages)
# finally, declare the list of builders. In this case, we only have one buil der # finally, declare the list of builders. In this case, we only have one buil der
c['builders'] = [ c['builders'] = [
util.BuilderConfig(name="simplebuild", workernames=['worker1', 'worker2' , 'worker3'], util.BuilderConfig(name="simplebuild", workernames=['worker1', 'worker2' , 'worker3'],
factory=f_simplebuild) factory=f_simplebuild)
] ]
So our builder is called ``simplebuild`` and can run on either of ``worker1``, ` So our builder is called ``simplebuild`` and can run on either of ``worker1``, `
`worker2`` and ``worker3``. `worker2`` or ``worker3``.
If our repository has other branches besides trunk, we could create another one If our repository has other branches besides trunk, we could create another one
or more builders to build them; in the example, only the checkout step would be or more builders to build them; in this example, only the checkout step would be
different, in that it would need to check out the specific branch. different, in that it would need to check out the specific branch.
Depending on how exactly those branches have to be built, the shell commands may be recycled, or new ones would have to be created if they are different in the branch. Depending on how exactly those branches have to be built, the shell commands may be recycled, or new ones would have to be created if they are different in the branch.
You get the idea. You get the idea.
The important thing is that all the builders be named differently and all be add ed to the ``c['builders']`` value (as can be seen above, it is a list of ``Build erConfig`` objects). The important thing is that all the builders be named differently and all be add ed to the ``c['builders']`` value (as can be seen above, it is a list of ``Build erConfig`` objects).
Of course the type and number of steps will vary depending on the goal; for exam ple, to just check that a commit doesn't break the build, we could include just up to the ``make all`` step. Of course the type and number of steps will vary depending on the goal; for exam ple, to just check that a commit doesn't break the build, we could include just up to the ``make all`` step.
Or we could have a builder that performs a more thorough test by also doing ``ma ke test`` or other targets. Or we could have a builder that performs a more thorough test by also doing ``ma ke test`` or other targets.
You get the idea. You get the idea.
Note that at each step except the very first we use ``haltOnFailure=True`` becau se it would not make sense to execute a step if the previous one failed (ok, it wouldn't be needed for the last step, but it's harmless and protects us if one d ay we add another step after it). Note that at each step except the very first we use ``haltOnFailure=True`` becau se it would not make sense to execute a step if the previous one failed (ok, it wouldn't be needed for the last step, but it's harmless and protects us if one d ay we add another step after it).
Schedulers Schedulers
---------- ----------
Now this is all nice and dandy, but who tells the builder (or builders) to run, and when? Now this is all nice and dandy, but who tells the builder (or builders) to run, and when?
This is the job of the `scheduler`, which is a fancy name for an element that wa its for some event to happen, and when it does, based on that information decide s whether and when to run a builder (and which one or ones). This is the job of the `scheduler` which is a fancy name for an element that wai ts for some event to happen, and when it does, based on that information, decide s whether and when to run a builder (and which one or ones).
There can be more than one scheduler. There can be more than one scheduler.
I'm being purposely vague here because the possibilities are almost endless and highly dependent on the actual setup, build purposes, source repository layout a nd other elements. I'm being purposely vague here because the possibilities are almost endless and highly dependent on the actual setup, build purposes, source repository layout a nd other elements.
So a scheduler needs to be configured with two main pieces of information: on on e hand, which events to react to, and on the other hand, which builder or builde rs to trigger when those events are detected. So a scheduler needs to be configured with two main pieces of information: on on e hand, which events to react to, and on the other hand, which builder or builde rs to trigger when those events are detected.
(It's more complex than that, but if you understand this, you can get the rest o f the details from the docs). (It's more complex than that, but if you understand this, you can get the rest o f the details from the docs).
A simple type of scheduler may be a periodic scheduler: when a configurable amou nt of time has passed, run a certain builder (or builders). A simple type of scheduler may be a periodic scheduler that runs a certain build er (or builders) when a configurable amount of time has passed.
In our example, that's how we would trigger a build every hour:: In our example, that's how we would trigger a build every hour::
from buildbot.plugins import schedulers from buildbot.plugins import schedulers
# define the periodic scheduler # define the periodic scheduler
hourlyscheduler = schedulers.Periodic(name="hourly", hourlyscheduler = schedulers.Periodic(name="hourly",
builderNames=["simplebuild"], builderNames=["simplebuild"],
periodicBuildTimer=3600) periodicBuildTimer=3600)
# define the available schedulers # define the available schedulers
c['schedulers'] = [hourlyscheduler] c['schedulers'] = [hourlyscheduler]
That's it. That's it.
Every hour this ``hourly`` scheduler will run the ``simplebuild`` builder. Every hour this ``hourly`` scheduler will run the ``simplebuild`` builder.
If we have more than one builder that we want to run every hour, we can just add If we have more than one builder that we want to run every hour, we can just add
them to the ``builderNames`` list when defining the scheduler and they will all them to the ``builderNames`` list when defining the scheduler.
be run. Or since multiple schedulers are allowed, other schedulers can be defined and ad
Or since multiple scheduler are allowed, other schedulers can be defined and add ded to ``c['schedulers']`` in the same way.
ed to ``c['schedulers']`` in the same way.
Other types of schedulers exist; in particular, there are schedulers that can be more dynamic than the periodic one. Other types of schedulers exist; in particular, there are schedulers that can be more dynamic than the periodic one.
The typical dynamic scheduler is one that learns about changes in a source repos itory (generally because some developer checks in some change), and triggers one or more builders in response to those changes. The typical dynamic scheduler is one that learns about changes in a source repos itory (generally because some developer checks in some change) and triggers one or more builders in response to those changes.
Let's assume for now that the scheduler "magically" learns about changes in the repository (more about this later); here's how we would define it:: Let's assume for now that the scheduler "magically" learns about changes in the repository (more about this later); here's how we would define it::
from buildbot.plugins import schedulers from buildbot.plugins import schedulers
# define the dynamic scheduler # define the dynamic scheduler
trunkchanged = schedulers.SingleBranchScheduler(name="trunkchanged", trunkchanged = schedulers.SingleBranchScheduler(name="trunkchanged",
change_filter=util.ChangeFil ter(branch=None), change_filter=util.ChangeFil ter(branch=None),
treeStableTimer=300, treeStableTimer=300,
builderNames=["simplebuild"] ) builderNames=["simplebuild"] )
# define the available schedulers # define the available schedulers
c['schedulers'] = [trunkchanged] c['schedulers'] = [trunkchanged]
This scheduler receives changes happening to the repository, and among all of th em, pays attention to those happening in "trunk" (that's what ``branch=None`` me ans). This scheduler receives changes happening to the repository, and among all of th em, pays attention to those happening in "trunk" (that's what ``branch=None`` me ans).
In other words, it filters the changes to react only to those it's interested in . In other words, it filters the changes to react only to those it's interested in .
When such changes are detected, and the tree has been quiet for 5 minutes (300 s econds), it runs the ``simplebuild`` builder. When such changes are detected, and the tree has been quiet for 5 minutes (300 s econds), it runs the ``simplebuild`` builder.
The ``treeStableTimer`` helps in those situations where commits tend to happen i n bursts, which would otherwise result in multiple build requests queuing up. The ``treeStableTimer`` helps in those situations where commits tend to happen i n bursts, which would otherwise result in multiple build requests queuing up.
What if we want to act on two branches (say, trunk and 7.2)? What if we want to act on two branches (say, trunk and 7.2)?
First we create two builders, one for each branch (see the builders paragraph ab ove), then we create two dynamic schedulers:: First, we create two builders, one for each branch, and then we create two dynam ic schedulers::
from buildbot.plugins import schedulers from buildbot.plugins import schedulers
# define the dynamic scheduler for trunk # define the dynamic scheduler for trunk
trunkchanged = schedulers.SingleBranchScheduler(name="trunkchanged", trunkchanged = schedulers.SingleBranchScheduler(name="trunkchanged",
change_filter=util.ChangeFil ter(branch=None), change_filter=util.ChangeFil ter(branch=None),
treeStableTimer=300, treeStableTimer=300,
builderNames=["simplebuild-t runk"]) builderNames=["simplebuild-t runk"])
# define the dynamic scheduler for the 7.2 branch # define the dynamic scheduler for the 7.2 branch
branch72changed = schedulers.SingleBranchScheduler( branch72changed = schedulers.SingleBranchScheduler(
name="branch72changed", name="branch72changed",
change_filter=util.ChangeFilter(branch='branches/7.2'), change_filter=util.ChangeFilter(branch='branches/7.2'),
treeStableTimer=300, treeStableTimer=300,
builderNames=["simplebuild-72"]) builderNames=["simplebuild-72"])
# define the available schedulers # define the available schedulers
c['schedulers'] = [trunkchanged, branch72changed] c['schedulers'] = [trunkchanged, branch72changed]
The syntax of the change filter is VCS-dependent (above is for SVN), but again o nce the idea is clear, the documentation has all the details. The syntax of the change filter is VCS-dependent (above is for SVN), but again, once the idea is clear, the documentation has all the details.
Another feature of the scheduler is that it can be told which changes, within th ose it's paying attention to, are important and which are not. Another feature of the scheduler is that it can be told which changes, within th ose it's paying attention to, are important and which are not.
For example, there may be a documentation directory in the branch the scheduler is watching, but changes under that directory should not trigger a build of the binary. For example, there may be a documentation directory in the branch the scheduler is watching, but changes under that directory should not trigger a build of the binary.
This finer filtering is implemented by means of the ``fileIsImportant`` argument to the scheduler (full details in the docs and - alas - in the sources). This finer filtering is implemented by means of the ``fileIsImportant`` argument to the scheduler (full details in the docs and - alas - in the sources).
Change sources Change sources
-------------- --------------
Earlier we said that a dynamic scheduler "magically" learns about changes; the f Earlier, we said that a dynamic scheduler "magically" learns about changes; the
inal piece of the puzzle are `change sources`, which are precisely the elements final piece of the puzzle is `change sources`, which are precisely the elements
in Buildbot whose task is to detect changes in the repository and communicate th in Buildbot whose task is to detect changes in a repository and communicate them
em to the schedulers. to the schedulers.
Note that periodic schedulers don't need a change source, since they only depend Note that periodic schedulers don't need a change source since they only depend
on elapsed time; dynamic schedulers, on the other hand, do need a change source on elapsed time; dynamic schedulers, on the other hand, do need a change source.
.
A change source is generally configured with information about a source reposito ry (which is where changes happen); a change source can watch changes at differe nt levels in the hierarchy of the repository, so for example it is possible to w atch the whole repository or a subset of it, or just a single branch. A change source is generally configured with information about a source reposito ry (which is where changes happen). A change source can watch changes at differe nt levels in the hierarchy of the repository, so for example, it is possible to watch the whole repository or a subset of it, or just a single branch.
This determines the extent of the information that is passed down to the schedul ers. This determines the extent of the information that is passed down to the schedul ers.
There are many ways a change source can learn about changes; it can periodically poll the repository for changes, or the VCS can be configured (for example thro ugh hook scripts triggered by commits) to push changes into the change source. There are many ways a change source can learn about changes; it can periodically poll the repository for changes, or the VCS can be configured (for example thro ugh hook scripts triggered by commits) to push changes into the change source.
While these two methods are probably the most common, they are not the only poss While these two methods are probably the most common, they are not the only poss
ibilities; it is possible for example to have a change source detect changes by ibilities. It is possible, for example, to have a change source detect changes b
parsing some email sent to a mailing list when a commit happens, and yet other m y parsing an email sent to a mailing list when a commit happens.
ethods exist. Yet other methods exist and the manual again has the details.
The manual again has the details.
To complete our example, here's a change source that polls a SVN repository ever y 2 minutes:: To complete our example, here's a change source that polls a SVN repository ever y 2 minutes::
from buildbot.plugins import changes, util from buildbot.plugins import changes, util
svnpoller = changes.SVNPoller(repourl="svn://myrepo/projects/coolproject", svnpoller = changes.SVNPoller(repourl="svn://myrepo/projects/coolproject",
svnuser="foo", svnuser="foo",
svnpasswd="bar", svnpasswd="bar",
pollinterval=120, pollinterval=120,
split_file=util.svn.split_file_branches) split_file=util.svn.split_file_branches)
skipping to change at line 239 skipping to change at line 239
We could have said:: We could have said::
repourl = "svn://myrepo/projects/coolproject/trunk" repourl = "svn://myrepo/projects/coolproject/trunk"
or:: or::
repourl = "svn://myrepo/projects/coolproject/branches/7.2" repourl = "svn://myrepo/projects/coolproject/branches/7.2"
to watch only a specific branch. to watch only a specific branch.
To watch another project, you need to create another change source -- and you ne To watch another project, you need to create another change source, and you need
ed to filter changes by project. to filter changes by project.
For instance, when you add a change source watching project 'superproject' to th For instance, when you add a change source watching project 'superproject' to th
e above example, you need to change:: e above example, you need to change the original scheduler from::
trunkchanged = schedulers.SingleBranchScheduler( trunkchanged = schedulers.SingleBranchScheduler(
name="trunkchanged", name="trunkchanged",
change_filter=filter.ChangeFilter(branch=None), change_filter=filter.ChangeFilter(branch=None),
# ... # ...
) )
to e.g.:: to e.g.::
trunkchanged = schedulers.SingleBranchScheduler( trunkchanged = schedulers.SingleBranchScheduler(
name="trunkchanged", name="trunkchanged",
change_filter=filter.ChangeFilter(project="coolproject", branch=None), change_filter=filter.ChangeFilter(project="coolproject", branch=None),
# ... # ...
) )
else coolproject will be built when there's a change in superproject. otherwise, coolproject will be built when there's a change in superproject.
Since we're watching more than one branch, we need a method to tell in which bra nch the change occurred when we detect one. Since we're watching more than one branch, we need a method to tell in which bra nch the change occurred when we detect one.
This is what the ``split_file`` argument does, it takes a callable that Buildbot will call to do the job. This is what the ``split_file`` argument does, it takes a callable that Buildbot will call to do the job.
The split_file_branches function, which comes with Buildbot, is designed for exa ctly this purpose so that's what the example above uses. The split_file_branches function, which comes with Buildbot, is designed for exa ctly this purpose so that's what the example above uses.
And of course this is all SVN-specific, but there are pollers for all the popula r VCSs. And of course this is all SVN-specific, but there are pollers for all the popula r VCSs.
But note: if you have many projects, branches, and builders it probably pays to not hardcode all the schedulers and builders in the configuration, but generate them dynamically starting from list of all projects, branches, targets etc. and using loops to generate all possible combinations (or only the needed ones, depe nding on the specific setup), as explained in the documentation chapter about :d oc:`../manual/customization`. Note that if you have many projects, branches, and builders, it probably pays no t to hardcode all the schedulers and builders in the configuration, but generate them dynamically starting from the list of all projects, branches, targets, etc , and using loops to generate all possible combinations (or only the needed ones , depending on the specific setup), as explained in the documentation chapter ab out :doc:`../manual/customization`.
Reporters Reporters
--------- ---------
Now that the basics are in place, let's go back to the builders, which is where the real work happens. Now that the basics are in place, let's go back to the builders, which is where the real work happens.
`Reporters` are simply the means Buildbot uses to inform the world about what's happening, that is, how builders are doing. `Reporters` are simply the means Buildbot uses to inform the world about what's happening, that is, how builders are doing.
There are many reporters: a mail notifier, an IRC notifier, and others. There are many reporters: a mail notifier, an IRC notifier, and others.
They are described fairly well in the manual. They are described fairly well in the manual.
One thing I've found useful is the ability to pass a domain name as the lookup a rgument to a ``mailNotifier``, which allows you to take an unqualified username as it appears in the SVN change and create a valid email address by appending th e given domain name to it:: One thing I've found useful is the ability to pass a domain name as the lookup a rgument to a ``mailNotifier``, which allows you to take an unqualified username as it appears in the SVN change and create a valid email address by appending th e given domain name to it::
from buildbot.plugins import reporter from buildbot.plugins import reporter
# if jsmith commits a change, mail for the build is sent to jsmith@example.o rg # if jsmith commits a change, an email for the build is sent to jsmith@examp le.org
notifier = reporter.MailNotifier(fromaddr="buildbot@example.org", notifier = reporter.MailNotifier(fromaddr="buildbot@example.org",
sendToInterestedUsers=True, sendToInterestedUsers=True,
lookup="example.org") lookup="example.org")
c['reporters'].append(notifier) c['reporters'].append(notifier)
The mail notifier can be customized at will by means of the ``messageFormatter`` argument, which is a class that Buildbot calls to format the body of the email, and to which it makes available lots of information about the build. The mail notifier can be customized at will by means of the ``messageFormatter`` argument, which is a class that Buildbot calls to format the body of the email, and to which it makes available lots of information about the build.
For more details, look into the :ref:`Reporters` section of the Buildbot manual. For more details, look into the :ref:`Reporters` section of the Buildbot manual.
Conclusion Conclusion
---------- ----------
Please note that this article has just scratched the surface; given the complexi ty of the task of build automation, the possibilities are almost endless. Please note that this article has just scratched the surface; given the complexi ty of the task of build automation, the possibilities are almost endless.
So there's much, much more to say about Buildbot. However, hopefully this is a p reparation step before reading the official manual. Had I found an explanation a s the one above when I was approaching Buildbot, I'd have had to read the manual just once, rather than multiple times. Hope this can help someone else. So there's much much more to say about Buildbot. Hopefully this has been a gentl e introduction before reading the official manual. Had I found an explanation as the one above when I was approaching Buildbot, I'd have had to read the manual just once, rather than multiple times. I hope this can help someone else.
(Thanks to Davide Brini for permission to include this tutorial, derived from on e he originally posted at http://backreference.org .) (Thanks to Davide Brini for permission to include this tutorial, derived from on e he originally posted at http://backreference.org .)
 End of changes. 22 change blocks. 
50 lines changed or deleted 47 lines changed or added

Home  |  About  |  Features  |  All  |  Newest  |  Dox  |  Diffs  |  RSS Feeds  |  Screenshots  |  Comments  |  Imprint  |  Privacy  |  HTTP(S)