"Fossies" - the Fresh Open Source Software Archive

Member "libisofs-1.5.4/doc/zisofs2_format.txt" (29 Oct 2020, 9701 Bytes) of package /linux/misc/libisofs-1.5.4.tar.gz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file.

    1 
    2                         Description of the zisofs2 Format
    3                               Revision 2.0-dev
    4 
    5                            as of zisofs2-tools by
    6                    Valentín KIVACHUK BURDÁ and Thomas SCHMITT
    7 
    8                                 1 Oct 2020
    9 
   10 
   11 The zisofs2 format was invented by Valentín KIVACHUK BURDÁ and 
   12 Thomas SCHMITT (as extension of zisofs by H. Peter Anvin). It compresses
   13 data file content, marks it by a header and provides a pointer array for 
   14 coarse random access. Within a RRIP enhanced ISO 9660 image the format 
   15 is additionally marked by a System Use entry with signature "ZF" or "Z2".
   16 
   17 The uncompressed size of a single zisofs2 compressed file is restricted
   18 to 2^64 - 1 bytes. Larger files shall not be compressed. 
   19 
   20 The format of version 1 of zisofs is supported by this specification.
   21 Using it for files with uncompressed size smaller than 4 GiB is friendly
   22 towards software which does not know about zisofs2.
   23 See section **LEGACY** for a summary of version 1 of zisofs.
   24 
   25 
   26 
   27                                 Data Types
   28 
   29 ISO 9660:7.3.1 - little endian 4-byte words
   30 ISO 9660:7.1.1 - unsigned single bytes
   31 ISO 9660:7.3.3 - 8-bytes value, first in little endian, then big endian.
   32 #uint64        - 8-bytes unsigned value in little endian
   33 
   34                                 Supported compressors
   35 
   36 The file header has this layout:
   37     @alg_id       @alg_char                Description
   38      1            'PZ' (50)(5A)            Zlib
   39      2            'XZ' (78)(7A)            XZ
   40      3            'L4' (6C)(34)            LZ4
   41      4            'ZD' (7A)(64)            Zstandard
   42      5            'B2' (62)(32)            Bzip2
   43 
   44 @alg_id is a 7.1.1 value. @alg_char is 2 ASCII characters stored as 2 bytes
   45 Values of @alg_id = 0 and @alg_char = 'pz'(70)(7A) are reserved and 
   46 must not be used. Other compressors are allowed and may be added to this 
   47 list in the future
   48 
   49                                 Compressor strategy
   50 
   51 The default strategy for a compressor is to compress each input data block 
   52 independently. The zisofs2 spec may define in the future other strategies,
   53 which will have a new @alg_id, @alg_char and a description in this section.
   54 
   55 
   56                                 File Header
   57 
   58 The file header has this layout:
   59 Offset     Type         Identifier    Contents
   60 --------------------------------------------------------------------------
   61   0        (8 bytes)    @hdr_magic    Magic num (EF 22 55 A1 BC 1B 95 A0)
   62   8        7.1.1        @hdr_version  File header version (0)
   63   9        7.1.1        @hdr_size     header_size >> 2 (6)
   64  10        7.1.1        @alg_id       Algorithm Type (>=1)
   65  11        7.1.1        @hdr_bsize    log2(block_size) (15, 16, or 17)
   66  12        #uint64      @size         Uncompressed file size
   67  20        (4 bytes)    -             Padding. Ignored
   68 
   69 So its size is 24.
   70 
   71 Readers shall be able to handle log2(block_size) values 15, 16 and 17
   72 i.e. block sizes 32 kB, 64 kB, and 128 kB. Writers must not use 
   73 other sizes.
   74 
   75                                Block Pointers
   76 
   77 There are ceil(input_size / block_size) input resp. output blocks.
   78 Each input block is of fixed size whereas the output blocks have varying
   79 size (down to 0). For each output block there is an offset pointer giving
   80 its byte address in the overall file content. The next block pointer in the
   81 array tells the start of the next block which begins immediately after the
   82 end of its predecessor. A final pointer (*eob*) gives the first invalid 
   83 byte address and thus marks the end of the last block.
   84 
   85 So there are ceil(input_size / block_size) + 1 block pointers.
   86 They are stored directly after the file header, i.e. beginning at byte 24, 
   87 as an array of values in #uint64 format (8 bytes).
   88 
   89 Legacy format (zisofs) may be used, which is described in section *LEGACY*
   90 
   91 
   92 
   93                                  Data Part
   94 
   95 The data part begins immediately after the pointer array (*eob*). In 
   96 principle it consists of the variable length output blocks as delivered by 
   97 different compression algorithms when fed with the fixed size input blocks.
   98 
   99 A special case of input and output block is defined:
  100 Zero-length blocks represent a block full of 0-bytes.
  101 Such input blocks do not get processed by compress2() but shall be mapped 
  102 to 0-sized output directly. Vice versa 0-sized blocks have to bypass 
  103 uncompress() when being read.
  104 
  105 
  106                          ZF System Use Entry Format
  107 
  108 The ZF entry follows the general layout of SUSP resp. RRIP.
  109 Its fields are:
  110 
  111   [1] "BP 1 to BP 2 - Signature Word" shall be (5A)(46) ("ZF").
  112 
  113   [2] "BP 3 - Length" shall specify as an 8-bit number the length in 
  114       bytes of the ZF entry recorded according to ISO 9660:7.1.1.
  115       This length is 16 decimal.
  116       Refer to **LEGACY**
  117 
  118   [3] "BP 4 - System Use Entry Version" shall be 2 as in ISO 9660:7.1.1.
  119       Refer to **LEGACY**
  120 
  121   [4] "BP 5 to BP 6 - Algorithm"  shall be two chars to indicate the
  122       compression algorithm. For example, (50)(5A) ("PZ")
  123       (This is a copy of @alg_char). Refer to **LEGACY**
  124 
  125   [5] "BP 7 - Header Size Div 4" shall specify as an 8-bit number the 
  126       number of 4-byte words in the header part of the file data recorded 
  127       according to ISO 9660:7.1.1.
  128       (This is a copy of @hdr_size).
  129 
  130   [6] "BP 8 - Log2 of Block Size" shall specify as an 8-bit number the 
  131       binary logarithm of the compression block size recorded according to
  132       ISO 9660:7.1.1.
  133       (This is a copy of header byte 13 (@hdr_bsize), resp. header BP 14.
  134        The value has to be 15, 16 or 17 i.e. 32 kiB, 64 kiB, or 128 kiB.)
  135 
  136   [7] "BP 9 to BP 16 - Virtual Uncompressed File Size" shall contain
  137       as a 64-bit unsigned little endian number the uncompressed 
  138       file size represented by the given extent. Refer to **LEGACY**
  139        
  140 
  141   | 'Z' | 'F' | LENGTH | 2 | 'P' | 'Z' | HEADER SIZE DIV 4 | 
  142   | LOG2 BLOCK SIZE | UNCOMPRESSED SIZE |
  143 
  144 Example (block size 128 kiB, uncompressed file size = 40 TB):
  145   { 'Z',  'F',  16,   2,  'P',  'Z',    8,   17,
  146    0x00, 0x80, 0xCA, 0x39, 0x61, 0x24, 0x00, 0x00 }
  147 
  148 
  149                          Z2 System Use Entry Format
  150 
  151 Linux kernels which are configured by CONFIG_ZISOFS to recognize zisofs
  152 but are not aware of zisofs2 will complain about ZF entries which announce
  153 algorithms other than "pz". The system log will show for each zisofs2
  154 compressed file at first stat(2) or open(2) a line like this:
  155 
  156   isofs: Unknown ZF compression algorithm: PZ
  157 
  158 To avoid these complaints, it is possible to use 
  159 
  160   [1] "BP 1 to BP 2 - Signature Word" shall be (5A)(32) ("Z2").
  161 
  162 instead of "ZF" with the System Use Entry Format specified above.
  163 Everything else shall be like in ZF format version 2, including the version
  164 number itself:
  165 
  166   [3] "BP 4 - System Use Entry Version" shall be 2 as in ISO 9660:7.1.1.
  167 
  168 
  169                                  **LEGACY**
  170 
  171 zisofs2 supports old readers by respecting the zisofs format. This section
  172 describes which definitions from zisofs2 must change to be compatible 
  173 with zisofs.
  174 
  175 - General behaviour
  176   The uncompressed size of a single zisofs compressed file is restricted
  177   to 4 GiB - 1. Larger files shall not be compressed.
  178 
  179 - Supported algorithms
  180   Only algorithm Zlib with default strategy is supported.
  181 
  182 - The file header must follow this structure:
  183 
  184   Offset   Type        Identifier    Contents
  185     0      (8 bytes)   @hdr_magic    Magic number (37 E4 53 96 C9 DB D6 07)
  186     8      7.3.1       @size         Uncompressed file size
  187    12      7.1.1       @hdr_size     header_size >> 2 (4)
  188    13      7.1.1       @hdr_bsize    log2(block_size) (15, 16, or 17)
  189    14      (2 bytes)   -             Reserved, must be zero
  190     
  191   So its size is 16.
  192 
  193 - Block pointers
  194   The array must use ISO 9660:7.3.1 (4 bytes) values.
  195 
  196 - ZF entry
  197 
  198 Its fields are:
  199 
  200   [1] "BP 1 to BP 2 - Signature Word" shall be (5A)(46) ("ZF").
  201 
  202   [2] "BP 3 - Length" must be 16 decimal.
  203 
  204   [3] "BP 4 - System Use Entry Version" must be 1.
  205 
  206   [4] "BP 5 to BP 6 - Algorithm" must be (70)(7A) ("pz").
  207 
  208   [5] "BP 7 - Header Size Div 4" - same as zisofs2.
  209 
  210   [6] "BP 8 - Log2 of Block Size" - same as zisofs2.
  211 
  212   [7] "BP 9 to BP 16 - Uncompressed Size" This field shall be recorded
  213       according to ISO 9660:7.3.3.
  214       (This number is the same as @size )
  215 
  216   | 'Z' | 'F' | LENGTH | 1 | 'p' | 'z' | HEADER SIZE DIV 4 | 
  217   | LOG2 BLOCK SIZE | UNCOMPRESSED SIZE |
  218 
  219 Example (block size 32 kiB, uncompressed file size = 1,234,567 bytes):
  220   { 'Z',  'F',   16,    1,  'p',  'z',    4,   15,
  221    0x87, 0xD6, 0x12, 0x00, 0x00, 0x12, 0xD6, 0x87 }
  222 
  223 
  224 References:
  225 
  226 zisofs2-tools
  227   https://github.com/vk496/zisofs2-tools
  228 
  229 zisofs-tools
  230   http://freshmeat.net/projects/zisofs-tools/
  231 
  232 zlib:
  233   /usr/include/zlib.h
  234 
  235 cdrtools with mkisofs
  236   ftp://ftp.berlios.de/pub/cdrecord/alpha
  237 
  238 ECMA-119 aka ISO 9660
  239   http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-119.pdf
  240 
  241 SUSP 1.12
  242   ftp://ftp.ymi.com/pub/rockridge/susp112.ps
  243 
  244 RRIP 1.12
  245   ftp://ftp.ymi.com/pub/rockridge/rrip112.ps
  246 
  247 zisofs version 1
  248   libisofs-*/doc/zisofs_format.txt
  249   https://dev.lovelyhq.com/libburnia/libisofs/raw/branch/master/doc/zisofs_format.txt
  250 
  251 ---------------------------------------------------------------------------
  252 
  253 This text is under
  254 Copyright (c) 2009 - 2010, 2020 Thomas SCHMITT <scdbackup@gmx.net>
  255 Copyright (c) 2020 - Valentín KIVACHUK BURDÁ <vk18496@gmail.com>
  256 It shall reflect the effective technical specifications as implemented in
  257 zisofs2-tools and the Linux kernel. So please contact mailing list
  258 <bug-xorriso@gnu.org> or to the copyright holders in private, if you
  259 want to make changes.
  260 Only if you cannot reach the copyright holder for at least one month it is
  261 permissible to modify and distribute this text under the license "GPLv3".
  262