"Fossies" - the Fresh Open Source Software Archive 
Member "libisofs-1.5.4/doc/zisofs2_format.txt" (29 Oct 2020, 9701 Bytes) of package /linux/misc/libisofs-1.5.4.tar.gz:
As a special service "Fossies" has tried to format the requested text file into HTML format (style:
standard) with prefixed line numbers.
Alternatively you can here
view or
download the uninterpreted source code file.
1
2 Description of the zisofs2 Format
3 Revision 2.0-dev
4
5 as of zisofs2-tools by
6 Valentín KIVACHUK BURDÁ and Thomas SCHMITT
7
8 1 Oct 2020
9
10
11 The zisofs2 format was invented by Valentín KIVACHUK BURDÁ and
12 Thomas SCHMITT (as extension of zisofs by H. Peter Anvin). It compresses
13 data file content, marks it by a header and provides a pointer array for
14 coarse random access. Within a RRIP enhanced ISO 9660 image the format
15 is additionally marked by a System Use entry with signature "ZF" or "Z2".
16
17 The uncompressed size of a single zisofs2 compressed file is restricted
18 to 2^64 - 1 bytes. Larger files shall not be compressed.
19
20 The format of version 1 of zisofs is supported by this specification.
21 Using it for files with uncompressed size smaller than 4 GiB is friendly
22 towards software which does not know about zisofs2.
23 See section **LEGACY** for a summary of version 1 of zisofs.
24
25
26
27 Data Types
28
29 ISO 9660:7.3.1 - little endian 4-byte words
30 ISO 9660:7.1.1 - unsigned single bytes
31 ISO 9660:7.3.3 - 8-bytes value, first in little endian, then big endian.
32 #uint64 - 8-bytes unsigned value in little endian
33
34 Supported compressors
35
36 The file header has this layout:
37 @alg_id @alg_char Description
38 1 'PZ' (50)(5A) Zlib
39 2 'XZ' (78)(7A) XZ
40 3 'L4' (6C)(34) LZ4
41 4 'ZD' (7A)(64) Zstandard
42 5 'B2' (62)(32) Bzip2
43
44 @alg_id is a 7.1.1 value. @alg_char is 2 ASCII characters stored as 2 bytes
45 Values of @alg_id = 0 and @alg_char = 'pz'(70)(7A) are reserved and
46 must not be used. Other compressors are allowed and may be added to this
47 list in the future
48
49 Compressor strategy
50
51 The default strategy for a compressor is to compress each input data block
52 independently. The zisofs2 spec may define in the future other strategies,
53 which will have a new @alg_id, @alg_char and a description in this section.
54
55
56 File Header
57
58 The file header has this layout:
59 Offset Type Identifier Contents
60 --------------------------------------------------------------------------
61 0 (8 bytes) @hdr_magic Magic num (EF 22 55 A1 BC 1B 95 A0)
62 8 7.1.1 @hdr_version File header version (0)
63 9 7.1.1 @hdr_size header_size >> 2 (6)
64 10 7.1.1 @alg_id Algorithm Type (>=1)
65 11 7.1.1 @hdr_bsize log2(block_size) (15, 16, or 17)
66 12 #uint64 @size Uncompressed file size
67 20 (4 bytes) - Padding. Ignored
68
69 So its size is 24.
70
71 Readers shall be able to handle log2(block_size) values 15, 16 and 17
72 i.e. block sizes 32 kB, 64 kB, and 128 kB. Writers must not use
73 other sizes.
74
75 Block Pointers
76
77 There are ceil(input_size / block_size) input resp. output blocks.
78 Each input block is of fixed size whereas the output blocks have varying
79 size (down to 0). For each output block there is an offset pointer giving
80 its byte address in the overall file content. The next block pointer in the
81 array tells the start of the next block which begins immediately after the
82 end of its predecessor. A final pointer (*eob*) gives the first invalid
83 byte address and thus marks the end of the last block.
84
85 So there are ceil(input_size / block_size) + 1 block pointers.
86 They are stored directly after the file header, i.e. beginning at byte 24,
87 as an array of values in #uint64 format (8 bytes).
88
89 Legacy format (zisofs) may be used, which is described in section *LEGACY*
90
91
92
93 Data Part
94
95 The data part begins immediately after the pointer array (*eob*). In
96 principle it consists of the variable length output blocks as delivered by
97 different compression algorithms when fed with the fixed size input blocks.
98
99 A special case of input and output block is defined:
100 Zero-length blocks represent a block full of 0-bytes.
101 Such input blocks do not get processed by compress2() but shall be mapped
102 to 0-sized output directly. Vice versa 0-sized blocks have to bypass
103 uncompress() when being read.
104
105
106 ZF System Use Entry Format
107
108 The ZF entry follows the general layout of SUSP resp. RRIP.
109 Its fields are:
110
111 [1] "BP 1 to BP 2 - Signature Word" shall be (5A)(46) ("ZF").
112
113 [2] "BP 3 - Length" shall specify as an 8-bit number the length in
114 bytes of the ZF entry recorded according to ISO 9660:7.1.1.
115 This length is 16 decimal.
116 Refer to **LEGACY**
117
118 [3] "BP 4 - System Use Entry Version" shall be 2 as in ISO 9660:7.1.1.
119 Refer to **LEGACY**
120
121 [4] "BP 5 to BP 6 - Algorithm" shall be two chars to indicate the
122 compression algorithm. For example, (50)(5A) ("PZ")
123 (This is a copy of @alg_char). Refer to **LEGACY**
124
125 [5] "BP 7 - Header Size Div 4" shall specify as an 8-bit number the
126 number of 4-byte words in the header part of the file data recorded
127 according to ISO 9660:7.1.1.
128 (This is a copy of @hdr_size).
129
130 [6] "BP 8 - Log2 of Block Size" shall specify as an 8-bit number the
131 binary logarithm of the compression block size recorded according to
132 ISO 9660:7.1.1.
133 (This is a copy of header byte 13 (@hdr_bsize), resp. header BP 14.
134 The value has to be 15, 16 or 17 i.e. 32 kiB, 64 kiB, or 128 kiB.)
135
136 [7] "BP 9 to BP 16 - Virtual Uncompressed File Size" shall contain
137 as a 64-bit unsigned little endian number the uncompressed
138 file size represented by the given extent. Refer to **LEGACY**
139
140
141 | 'Z' | 'F' | LENGTH | 2 | 'P' | 'Z' | HEADER SIZE DIV 4 |
142 | LOG2 BLOCK SIZE | UNCOMPRESSED SIZE |
143
144 Example (block size 128 kiB, uncompressed file size = 40 TB):
145 { 'Z', 'F', 16, 2, 'P', 'Z', 8, 17,
146 0x00, 0x80, 0xCA, 0x39, 0x61, 0x24, 0x00, 0x00 }
147
148
149 Z2 System Use Entry Format
150
151 Linux kernels which are configured by CONFIG_ZISOFS to recognize zisofs
152 but are not aware of zisofs2 will complain about ZF entries which announce
153 algorithms other than "pz". The system log will show for each zisofs2
154 compressed file at first stat(2) or open(2) a line like this:
155
156 isofs: Unknown ZF compression algorithm: PZ
157
158 To avoid these complaints, it is possible to use
159
160 [1] "BP 1 to BP 2 - Signature Word" shall be (5A)(32) ("Z2").
161
162 instead of "ZF" with the System Use Entry Format specified above.
163 Everything else shall be like in ZF format version 2, including the version
164 number itself:
165
166 [3] "BP 4 - System Use Entry Version" shall be 2 as in ISO 9660:7.1.1.
167
168
169 **LEGACY**
170
171 zisofs2 supports old readers by respecting the zisofs format. This section
172 describes which definitions from zisofs2 must change to be compatible
173 with zisofs.
174
175 - General behaviour
176 The uncompressed size of a single zisofs compressed file is restricted
177 to 4 GiB - 1. Larger files shall not be compressed.
178
179 - Supported algorithms
180 Only algorithm Zlib with default strategy is supported.
181
182 - The file header must follow this structure:
183
184 Offset Type Identifier Contents
185 0 (8 bytes) @hdr_magic Magic number (37 E4 53 96 C9 DB D6 07)
186 8 7.3.1 @size Uncompressed file size
187 12 7.1.1 @hdr_size header_size >> 2 (4)
188 13 7.1.1 @hdr_bsize log2(block_size) (15, 16, or 17)
189 14 (2 bytes) - Reserved, must be zero
190
191 So its size is 16.
192
193 - Block pointers
194 The array must use ISO 9660:7.3.1 (4 bytes) values.
195
196 - ZF entry
197
198 Its fields are:
199
200 [1] "BP 1 to BP 2 - Signature Word" shall be (5A)(46) ("ZF").
201
202 [2] "BP 3 - Length" must be 16 decimal.
203
204 [3] "BP 4 - System Use Entry Version" must be 1.
205
206 [4] "BP 5 to BP 6 - Algorithm" must be (70)(7A) ("pz").
207
208 [5] "BP 7 - Header Size Div 4" - same as zisofs2.
209
210 [6] "BP 8 - Log2 of Block Size" - same as zisofs2.
211
212 [7] "BP 9 to BP 16 - Uncompressed Size" This field shall be recorded
213 according to ISO 9660:7.3.3.
214 (This number is the same as @size )
215
216 | 'Z' | 'F' | LENGTH | 1 | 'p' | 'z' | HEADER SIZE DIV 4 |
217 | LOG2 BLOCK SIZE | UNCOMPRESSED SIZE |
218
219 Example (block size 32 kiB, uncompressed file size = 1,234,567 bytes):
220 { 'Z', 'F', 16, 1, 'p', 'z', 4, 15,
221 0x87, 0xD6, 0x12, 0x00, 0x00, 0x12, 0xD6, 0x87 }
222
223
224 References:
225
226 zisofs2-tools
227 https://github.com/vk496/zisofs2-tools
228
229 zisofs-tools
230 http://freshmeat.net/projects/zisofs-tools/
231
232 zlib:
233 /usr/include/zlib.h
234
235 cdrtools with mkisofs
236 ftp://ftp.berlios.de/pub/cdrecord/alpha
237
238 ECMA-119 aka ISO 9660
239 http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-119.pdf
240
241 SUSP 1.12
242 ftp://ftp.ymi.com/pub/rockridge/susp112.ps
243
244 RRIP 1.12
245 ftp://ftp.ymi.com/pub/rockridge/rrip112.ps
246
247 zisofs version 1
248 libisofs-*/doc/zisofs_format.txt
249 https://dev.lovelyhq.com/libburnia/libisofs/raw/branch/master/doc/zisofs_format.txt
250
251 ---------------------------------------------------------------------------
252
253 This text is under
254 Copyright (c) 2009 - 2010, 2020 Thomas SCHMITT <scdbackup@gmx.net>
255 Copyright (c) 2020 - Valentín KIVACHUK BURDÁ <vk18496@gmail.com>
256 It shall reflect the effective technical specifications as implemented in
257 zisofs2-tools and the Linux kernel. So please contact mailing list
258 <bug-xorriso@gnu.org> or to the copyright holders in private, if you
259 want to make changes.
260 Only if you cannot reach the copyright holder for at least one month it is
261 permissible to modify and distribute this text under the license "GPLv3".
262