"Fossies" - the Fresh Open Source Software Archive

Member "apt-1.8.4/doc/acquire-additional-files.md" (19 Sep 2019, 16052 Bytes) of package /linux/misc/apt-1.8.4.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format (assuming markdown format). Alternatively you can here view or download the uninterpreted source code file. A member file download can also be achieved by clicking within a package contents listing on the according byte size field. See also the last Fossies "Diffs" side-by-side code changes report for "acquire-additional-files.md": 1.8.2_vs_1.9.2.

Acquire additional files in ‘update’ operations

The download and verification of data from multiple sources in different compression formats, with partial downloads and patches is an involved process which is hard to implement correctly and securely.

APT front-ends share the code and binaries to make this happen in libapt with the Acquire system, supported by helpers shipped in the apt package itself and additional transports in individual packages like apt-transport-https.

For its own operation libapt needs or can make use of Packages, Sources and Translation- files, which it will acquire by default, but a repository might contain more data files (e.g. Contents) a front-end (e.g. apt-file) might want to use and would therefore need to be downloaded as well.

This file describes the configuration scheme such a front-end can use to instruct the Acquire system to download those additional files.

The Configuration Stanza

The Acquire system uses the same configuration settings to implement the files it downloads by default. These settings are the default, but if they would be written in a configuration file the configuration instructing the Acquire system to download the Packages files would look like this (see also apt.conf(5) manpage for configuration file syntax):

Acquire::IndexTargets::deb::Packages {
    MetaKey "$(COMPONENT)/binary-$(ARCHITECTURE)/Packages";
    ShortDescription "Packages";
    Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Packages";

    flatMetaKey "Packages";
    flatDescription "$(RELEASE) Packages";

    Optional "no";
};

All files which should be downloaded (nicknamed Targets) are mentioned below the Acquire::IndexTargets scope. deb is here the type of the sources.list entry the file should be acquired for. The only other supported value is hence deb-src. Beware: You can’t specify multiple types here and you can’t download the same (evaluated) MetaKey from multiple types!

After the type you can pick any valid and unique string which preferable refers to the file it downloads (In the example we picked Packages). This string is used as identifier (if not explicitly set otherwise) for the target class and accessible as Identifier and Created-By e.g. in the apt-get indextargets output as detailed below. The identifier is also used to allow user to enable/disable targets per sources.list entry.

All targets have three main properties you can define:

Additional optional properties:

The acquire system will automatically choose to download a compressed file if it is available and uncompress it for you, just as it will also use PDiff patching if provided by the repository and enabled by the user. You only have to ensure that the Release file contains the information about the compressed files/PDiffs to make this happen. NO properties have to be set to enable this!

More properties exist, but these should NOT be set by front-ends requesting files. They exist for internal and end-user usage only. Some of these are – which are documented here only to ensure that they aren’t accidentally used by front-ends:

More examples

The stanzas for Translation-* files as well as for Sources files would look like this:

Acquire::IndexTargets { deb::Translations { MetaKey “(COMPONENT)/i18n/Translation(LANGUAGE)”; ShortDescription “Translation-$(LANGUAGE)"; Description "$(RELEASE)/(COMPONENT)Translation(LANGUAGE)”;

    flatMetaKey "$(LANGUAGE)";
    flatDescription "$(RELEASE) Translation-$(LANGUAGE)";
};

deb-src::Sources {
    MetaKey "$(COMPONENT)/source/Sources";
    ShortDescription "Sources";
    Description "$(RELEASE)/$(COMPONENT) Sources";

    flatMetaKey "Sources";
    flatDescription "$(RELEASE) Sources";

    Optional "no";
};

};

Substitution variables

As seen in the examples, properties can contain placeholders filled in by the acquire system. The following variables are known; note that unknown variables have no default value nor are they touched: They are printed as-is.

Note that while more variables might exist in the implementation, these are to be considered undefined and their usage strongly discouraged. If you have a need for other variables contact us.

Accessing files

Do NOT hardcode specific file locations, names or compression types in your application! You will notice that the configuration options give you no choice over where the downloaded files will be stored. This is by design so multiple applications can download and use the same file rather than each and every one of them potentially downloads and uses its own copy somewhere on disk.

apt-get indextargets can be used to get the location as well as other information about all files downloaded (aka: you will see Packages, Sources and Translation- files here as well). Provide a line of the default output format as parameter to filter out all entries which do not have such a line. With --format, you can further more define your own output style. The variables are what you see in the output, just all uppercase and wrapped in $(), as in the configuration file.

To get all the filenames of all Translation-en files you can e.g. call:

apt-get indextargets --format '$(FILENAME)' "Identifier: Translations" "Language: en"

The line-based filtering and the formatting is rather crude and feature- less by design: The default format is Debian’s standard format deb822 (in particular: Field names are case-insensitive and the order of fields in the stanza is undefined), so instead of apt reimplementing powerful filters and formatting for this command, it is recommend to use piping and dedicated tools like grep-dctrl if you need more than the basics provided.

Accessing this information via libapt is done by reading the sources.lists (pkgSourceList), iterating over the metaIndex objects this creates and calling GetIndexTargets() on them. See the source code of apt-get indextargets for a complete example.

Note that by default targets are not listed if they weren’t downloaded. If you want to see all targets, you can use the --no-release-info, which also removes the Codename, Suite, Version, Origin, Label and Trusted fields from the output as these also display data which needs to be downloaded first and could hence be inaccurate [on the pro-side: This mode is faster as it doesn’t require a valid binary cache to operate]. The most notable difference perhaps is in the Filename field through: By default it indicates an existing file, potentially compressed (Hint: libapt users can use FileFd to open compressed files transparently). In the --no-release-info mode the indicated file doesn’t need to exist and it will always refer to an uncompressed file, even if the index would be (or is) stored compressed.

Remarks on fields only available in (default) --release-info mode:

Remarks on other available fields:

Again, additional fields might be visible in certain implementations, but you should avoid using them and instead talk to us about a portable implementation.

Multiple applications requiring the same files

It is highly encouraged that applications talk to each other and to us about which files they require. It is usually best to have a common package ship the configuration needed to get the files, but specific needs might require specific solutions. Again: talk to us.

Bad things will happen if multiple front-ends request the same file(s) via different targets, which is another reason why coordination is very important!

Acquiring files not mentioned in the Release file

You can’t. This is by design as these files couldn’t be verified to not be modified in transit, corrupted by the download process or simple if they are present at all on the server, which would require apt to probe for them. APT did this in the past for legacy reasons, we do not intend to go back to these dark times.

This is also why you can’t request files from a different server. It would have the additional problem that this server might not even be accessible (e.g. proxy settings) or that local sources (file:/, cdrom:/) start requesting online files…

In other words: We would be opening Pandora’s box.

Acquiring files to a specific location on disk

You can’t by design to avoid multiple front-ends requesting the same file to be downloaded to multiple different places on (different) disks (among other reasons). See the next point for a solution if you really have to force a specific location by creating symlinks.

Post processing the acquired files

You can’t modify the files apt has downloaded as apt keeps state with e.g. the modification times of the files and advanced features like PDiffs break.

You can however install an APT::Update::Post-Invoke{-Success,} hook script and use them to copy (modified) files to a different location. Use apt-get indextargets (or similar) to get the filenames – do not look into /var/lib/apt/lists directly!

Please avoid time consuming calculations in the scripts and instead just trigger a background task as there is little to no feedback for the user while hook scripts run.