"Fossies" - the Fresh Open Source Software Archive

Member "firefox-69.0.1/taskcluster/docs/partner-repacks.rst" (17 Sep 2019, 11538 Bytes) of package /linux/www/firefox-69.0.1.source.tar.xz:


As a special service "Fossies" has tried to format the requested source page into HTML format (assuming markdown format). Alternatively you can here view or download the uninterpreted source code file. A member file download can also be achieved by clicking within a package contents listing on the according byte size field.

Partner repacks

We create slightly-modified Firefox releases for some extra audiences

We use the phrase "partner repacks" to refer to all these builds because they use the same process of repacking regular Firefox releases with additional files. The specific differences depend on the type of build.

We produce partner repacks for some beta builds, and for release builds, as part of the release automation. We don't produce any files to update these builds as they are handled automatically (see updates).

Parameters & Scheduling

Partner repacks have a number of parameters which control how they work:

We split the repacks into two 'paths', EME-free and everything else, to retain some flexibility over enabling/disabling them separately. This costs us some duplication of the kinds in the repacking stack. The two enable parameters are booleans to turn these two paths on/off. We set them in release-runner3's is_partner_enabled() <https://dxr.mozilla .org/build-central/search?q=function%3Ais_partner_enabled&redirect=true> when starting a release. They're both true for Firefox betas >= b8 and releases, but otherwise disabled.

release_partner_config is a dictionary of configuration data which drives the task generation logic. It's usually looked up during the release promotion action task, using the Github GraphQL API in the get_partner_config_by_url() function, with the url defined in taskcluster/ci/config.yml <https://dxr.mozilla .org/mozilla-release/search?q=regexp%3A^partner+path%3Aconfig.yml&redirect=true>.

release_partner_build_number is an integer used to create unique upload paths in the firefox candidates directory, while release_partners is a list of partners that should be repacked (i.e. a subset of the whole config). Both are intended for use when respinning a few partners after the regular Firefox has shipped. More information on that can be found in the release-warrior docs <https://github.com/mozilla-releng/releasewarrior-2 .0/blob/master/docs/misc-operations/off-cycle-partner-repacks -and-funnelcake.md>.

Most of the machine time for generating partner repacks takes place in the promote phase of the automation, or promote_rc in the case of X.0 release candidates. The EME-free builds are copied into the Firefox releases directory in the push phase, along with the regular bits.

Configuration

We need some configuration to know what to repack, and how to do that. The what is defined by default.xml manifests, as used with the repo tool for git. The default.xml for EME-free <https://github .com/mozilla-partners/mozilla-EME-free-manifest/blob/master/default.xml> illustrates this:

<?xml version="1.0" ?>
<manifest>
  <remote fetch="git@github.com:mozilla-partners/" name="mozilla-partners"/>
  <remote fetch="git@github.com:mozilla/" name="mozilla"/>

  <project name="repack-scripts" path="scripts" remote="mozilla-partners" revision="master"/>
  <project name="build-tools" path="scripts/tools" remote="mozilla" revision="master"/>
  <project name="mozilla-EME-free" path="partners/mozilla-EME-free" remote="mozilla-partners" revision="master"/>
</manifest>

The repack-scripts and build-tools repos are found in all manifests, and then there is a list of partner repositories which contain the how configuration. Some of these repos are not publicly visible.

A partner repository may contain multiple configurations inside the desktop directory. Each subdirectory must contain a repack.cfg and a distribution directory, the latter containing the customizations needed. Here's EME-free's repack.cfg:

aus="mozilla-EMEfree"
dist_id="mozilla-EMEfree"
dist_version="1.0"
linux-i686=false
linux-x86_64=false
locales="ach af an ar"  # truncated for display here
mac=true
win32=true
win64=true
output_dir="%(platform)s-EME-free/%(locale)s"

# Upload params
upload_to_candidates=true

Note the list of locales and boolean toggles for enabling platforms. The output_dir and upload_to_candidates parameters are only present for repacks which are uploaded into the candidates directory.

All customizations will be placed in the distribution directory at the root of the Firefox install directory, or in the case of OS X in Firefox.app/Contents/Resources/distribution/. A distribution.ini file is the minimal requirement, here's an example from EME-free <https://github.com/mozilla-partners/mozilla-EME-free/blob/master/desktop/mozilla-EME-free/distribution /distribution.ini>:

# Partner Distribution Configuration File
# Author: Mozilla
# Date: 2015-03-27

[Global]
id=mozilla-EMEfree
version=1.0
about=Mozilla Firefox EME-free

[Preferences]
media.eme.enabled=false
app.partner.mozilla-EMEfree="mozilla-EMEfree"

Extensions and other customizations might also be included in repacks.

Repacking process

The stack of tasks to create partner repacks is broadly similar to localised nightlies and regular releases. The basic form is

Some key divergences are:

Partner repack

There is one task per platform in this step, calling out to scripts/desktop_partner_repacks.py <https://hg.mozilla.org/releases/mozilla-release/file/default/testing/mozharness/scripts /desktop_partner_repacks.py> in mozharness to prepare an environment and then perform the repacks.

It takes as input the build-signing and l10n-signing artifacts, which are all zip/tar.gz/tar.bz2 archives, simplifying the repack process by avoiding dmg and exe. Windows produces target.zip & setup.exe, Mac is target.tar.gz, Linux is the final product target.tar.bz2 (beetmover handles pretty naming as usual).

Signing

We chunk the single partner repack task out to a signing task per artifact at this point. For example, EME-free will become ~95 tasks, one for each locale. We collect the target.tar.gz from the upstream, and return a signed target.tar.gz. We use a target.dmg artifact for nightlies/regular releases, but this is converted to target.tar.gz by the signing scriptworker before sending it to the signing server, so partners are equivalent.

Repackage

Mac has a repackage job for each of the signing tasks. Windows repackages are chunked here to the same granularity as mac. Takes target.zip & setup.exe to produce target.exe on Windows, and target.tar.gz to produce target.dmg on Mac. There's no need to produce any complete.mar files here like regular release bits do because we can reuse those.

Chunking dummy

We're need Linux chunked at the next step so this dummy takes care of that for the relatively simple path Linux follows. One task per sub config+locale combination, the same as Windows and Mac. This doesn't need to exist for EME-free because we don't need to create Linux builds there.

Repackage Signing

This step GPG signs all platforms, and sha2signcode signs the Windows installer.

Beetmover

Moves and renames the artifacts to their public location in the candidates directory, or a private S3 bucket. Each task will have the project:releng:beetmover:action:push-to-partner scope, with public uploads having project:releng:beetmover:bucket:release and private uploads using project:releng:beetmover:bucket:partner. The upload_to_candidates key in the partner config controls the second scope. There's a separate partner code path in beetmoverscript <https://github .com/mozilla-releng/beetmoverscript>.

Beetmover checksums

The EME-free builds should be present in our SHA256SUMS file and friends (e.g. <https://archive .mozilla.org/pub/firefox/releases/61.0/SHA256SUMS>) so we beetmove the target.checksums from the beetmover tasks into the candidates directory. They get picked up by the release-generate-checksums kind.

Updates

It's very rare to need to update a partner repack differently from the original release build but we retain that capability. A partner build with distribution name foo, based on a release Firefox build, will query for an update on the release-cck-foo channel. If the update server Balrog finds no rule for that channel it will fallback to the release channel. The update files for the regular releases do not modify the distribution/ directory, so the customizations are not modified.

Bug 1430254 is an example of an exception to this logic.