"Fossies" - the Fresh Open Source Software Archive
Member "lrzip-0.641/ChangeLog" (15 Feb 2021, 48853 Bytes) of package /linux/misc/lrzip-0.641.tar.xz:
As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard
) with prefixed line numbers.
Alternatively you can here view
the uninterpreted source code file.
See also the last Fossies "Diffs"
side-by-side code changes report for "ChangeLog": 0.631_vs_0.640
1 lrzip ChangeLog
3 NOVEMBER 2019, updates to version 0.631, Peter Hyman
5 * Fixups to Assembler code in configure.ac lzma/C/Makefile.am by using
6 optimized code from p7zip LZMA SDK 16.02. Now works for 64 and 32 bit using
7 one optimized source. Hack to allow libtool to compile and link assembler
11 JUNE 2016, version 0.630 Con Kolivas, Peter Hyman, Petr Písař, Joel Fredrikson
13 * checksum.buf should only be changed after the semaphore wait
14 * Update README
15 * Add documentation for recursive mode
16 * Implement gzip compatible -r recursive option
17 * Add initial argument processing for recursive option
18 * Tidy
19 * Add one more verbose for compat mode
20 * Add support for various combinations in compat mode
21 * models is array of chars. char's signess is implementation specific. It's
22 unsigned on ARMv7. Unsigned char cannot represent negative values. GCC 6
23 complains about it:
24 * Fix decompression of multiple chunk encrypted archives
25 * Tidy gotos
26 * Show correct lengths during testing on big endian and compressed archives
27 * Update copyright dates
28 * Allow less than maxram to be malloced for checksum to fix Failed to malloc
29 ckbuf in hash_search2
30 * Base temporary output buffer on maximum mallocable, not maxram
31 * Enable subdir objects for future automake compatibility
32 * Add support for -m option in lrztar
33 * Big endian fix for Solaris Sparc courtesy of joelfredrikson.
34 * Fixed typographical error, changed accomodate to accommodate in README.
35 * A whitespace fix on lrztar.
36 * Add sanity check to prevent trying to malloc more ram than a
37 system/environment is capable of
38 * Cosmetic help change for compat
39 * Add rudimentary manpage for lrz
40 * Fix lrz symbolic linkage
41 * Do not fail if we are unable to write temporary files, giving a warning only
42 that it might fail if we don't have enough ram
43 * Try /tmp/ if none of the temporary environment directories or the current
44 directory are writeable
45 * Set STDOUT correctly in compat mode
46 * Style police
47 * Fix false warning on decompressing from stdin without keep files
48 * Fix false warning on compressing from stdin without keep files
49 * Don't show extra message in compat mode decompress
50 * Show correct appname when called in compat mode
51 * Add support for progress, fast and best flags in compat mode
52 * Add compatibility mode with gzip when called as lrz
53 * Correct adding slash to control->tmpdir. off-by-one error.
54 * Update manpage for long options
57 MARCH 2015, version 0.621 Con Kolivas, Peter Hyman
59 * Fix libzpaq.cpp warnings
60 * Fix warnings in LzmaLib.c
61 * Making the high buffer only one page size is faster for sliding mmap mode
62 * Fix incompatible log callback in liblrzip
63 * Use PRId64 instead of lld in fscanf
64 * Use int64_t for i64
65 * Fix ISO C warning in lrzip.c
66 * Fix ISO C warning in lrzip.c
67 * Fix ISO C warning in runzip.c
68 * Fix ISO C warnings in rzip.c
69 * Fix iso c warning in util.c
70 * Fix EOL to unix on libzpaq
71 * control->tmpdir fixes
72 * Null terminate long options to find unhandled options
73 * Trivial date
74 * Add long option support
75 * Brace failure lead to broken dump to stdout
76 * fflush messages on print_err
77 * Fix inverse logic
78 * Honour the proper unix environment variable for temporary directory and then
79 try variations
80 * Attempt to decompress file in ram only if we are unable to create temporary
81 files when decompressing to stdout
82 * Avoid checking return values repeatedly on compress that only lead to failures
84 * Microoptimisation
85 * Don't check for failure condition that can no longer occur
86 * Don't check twice for failure conditions that are fatal in hot paths
87 * Cache the chunk bytes value to avoid setting it on each read_header call
88 * fake_mremap is only used when defined to mremap
89 * Remove unused cksem functions
90 * Fix remaining use of mutexes lock/unlocking in different threads with cksems,
91 corecting cksem usage on osx
92 * Update copyright dates
93 * Make match_len a function completely removing all indirect calls to get_sb,
94 significantly speeding up the single_get_sb case
95 * Make full_tag a pointer allowing us to avoid a function call for get_sb
96 * Call sliding_get_sb directly in sliding_next_tag
97 * Make next_tag a pointer to allow ordinary mapping to avoid an extra function
99 * fix order of lrzip.conf search
102 MARCH 2015, version 0.620 Con Kolivas, Peter Hyman, Chris Spiegel, Elmir Jagudin
104 * Increase maxram when we abandon use of temporary input/output buffers
105 * Don't delete the tmpinfile when decompressing from stdin before allowing seek
106 to end to succeed in checking md5
107 * Use temporary file from read_seekto when STDIN will not fit in the ram input
109 * Remove unused read_i64 function
110 * Add message about issue tracker in BUGS
111 * Use a common exit path iin lrzip_compress/decompress and fix lr leak on
112 successful return
113 * Fix parenthesis placement inside of unlikely().
114 * Clear sa_mask and sa_handler before calling sigaction().
115 * Fix for lrzip -i. Decompressed size wrong
116 * added '-m' command line option
117 * Fix wrong README file being included in Makefile
118 * Pass strict sizes to decompress length, rounding up only the amount we're
119 allocating to not confuse decompression libraries
120 * Convert the thread locking to use cksems
121 * Add cksems to util.h
122 * Fix 'Failed to malloc ckbuf in hash_search2' with very large files.
123 * Round up compression and decompression buffers to page size since malloc will
124 allocate them that large anyway.
125 * Increase the compressed buffer size given to libzpaq in case of incompressible
126 data since it does not check if it's trying to write beyond the end of the
128 * Provide a helper function to round a value up to the nearest page size for
129 malloc optimisations.
132 SEPTEMBER 2013, version 0.616 George Makrydakis
134 * Making things more homogeneous in checks, some space removed.
135 * Simplifying and refactoring logic, alignment edits, empty semantic rewire.
136 * Target directories must always exist and -f should not create them.
137 * Deferring output directory after -f check has taken place.
138 * Preserve extraction semantics without resorting to tar stripping.
139 * Making sure last forward slash is removed from input path.
140 * Removing pointless forced overwrite point check.
143 SEPTEMBER 2013, version 0.615 Con Kolivas, George Makrydakis
145 * Adjusting -O flag semantics, options help update.
146 * Making -O flag operative for lrztar, whitespace path fix, lrzuntar fix, other
148 * Further tighten up ram restrictions with stdin/stdout to prevent running out
149 of memory with all the buffers involved.
150 * Massive files fail with -U due to trying to allocate the whole lot in ram
151 while doing checksums. Do it piecemeal to avoid the problem. Patch and debugging
152 courtesy of Adam Tkáč.
153 * We have to run through the clear buffer function even for empty buffers or
154 corrupt archives with empty match streams.
155 * MD5 code uses little endian so remove arbitrary SWAP macro and explicitly use
157 * Rewrite the magic if we receive lzma properties and have not yet written them
158 yet during stdout operation.
159 * Set the control lzma properties only once.
160 * Add a control lock mutex for protecting certain control variables.
161 * Fix stdin fake mremap creating null bytes on osx, patch courtesy of John
163 * Cache frequently used indirectly referenced variables in the sliding mmap
165 * Micro-optimise sliding_get_sb_range.
166 * A fix for a bug where large files containing the same non-zero bytes which
167 requires a sliding window, courtesy of Serge Belyshev.
168 * Put vchar should take a 64 bit integer and is used from more than one call
169 site so uninline it.
170 * Microoptimise in hash_search.
171 * Inline rzip functions used from only one caller.
172 * Check for successful calloc of hash table only after performing it.
173 * stdin_eof is just a bool
174 * hash_bits can only be up to 64 so use a char type.
175 * Check endianness of build with autotools to enable md5 support on more
179 JULY 2012, version 0.614 Con Kolivas
180 * Fix lrztar failing due to incorrect fix for -S freeing segfault.
183 JULY 2012, version 0.613 Con Kolivas, Serge Belyshev.
184 * Fix other non-win32 compilation for libzpaq.
185 * Fix libzpaq compilation on other non-win32 OSs.
186 * Fix Types.h missing which breaks typedefs on macosx.
187 * Cope with missing ffsll() on various platforms.
188 * Fix get_ram compilation on osx.
189 * Rename lrzip.h to lrzip_core.h to cope with case insensitive filesystems not
190 working with Lrzip.h vs lrzip.h
191 * Include test for pthread.h and include header in lrzip_private.h.
192 * Use system extensions in configure.ac to allow to build on non-gnu-linux
194 * Fix invalid free error when -o and -S are used together.
195 * Use no back end compression for blocks smaller than 64 bytes to avoid issues
196 to do with minimum buffer size and minimum match sizes of the rzip
198 * Fix md5 calculaton on blocks larger then 2^32. Bug reproduced and debugged by
199 Serge Belyshev. Patch by Paul Eggert.
202 MARCH 2012, version 0.612 Con Kolivas
203 * Minor formatting improvement to delete misaligned %.
204 * Implement progress counter on zpaq compress/decompress per thread within the
205 Reader function.
206 * Add some lrzip style defines and optimisations to the lrzip functions in
208 * Re-instantiate the lzo compresses test for zpaq compression.
209 * Remove redundant code.
210 * Delete now-unused zpipe code.
211 * Implement virtual write support for bufWrite to speed up zpaq support and
212 simplify the virtual read function for bufRead.
213 * Add virtual read() support to the bufRead class to speed up encoding/decoding
214 via the zpaq library.
215 * Use libzpaq backend for decompression.
216 * Move zpaq compression to new libzpaq library back end.
217 * Merge branch 'master' into zpaq
218 * Add minimal libzpaq compilation support.
219 * Move to libzpaq directory.
220 * Import libzapq files.
221 * Fix Lrzip.h not being installed into include directory and make distcheck
225 MARCH 2012, version 0.611 Con Kolivas.
226 * Move the update counter to earlier in the hash_search so it still updates even
227 when there are not matches.
228 * Perform all checksumming in a separate thread to speed up the hash search in
229 the rzip phase.
230 * Optimise the hot function empty_hash.
231 * Micro-optimise the hot next_tag function.
232 * Micro-optimise the match_len function.
233 * Pass one less variable to the get_sb function as an optimisation.
234 * Cache the hash_entry value passing one less variable to hash_empty as a
236 * Use ffsl for a faster lesser_bitness function.
237 * Prevent lrzuntar from automatically overwriting existing directories without
238 the -f option being specified.
239 * Calling lrzip via lrzcat would fail as the proper parameters weren't being
240 passed to the control structure. Fix.
243 MARCH 2012, version 0.610 Con Kolivas, Michael Blumenkrantz
244 * Implement complete set of liblrzip libraries, documentation and example uses
245 with support for simple lrzip_compress() and lrzip_decompress() or complete
246 fine-grained control over all compression and decompression options.
247 * Use as much of the low buffer as possible with a single memcopy before going
248 fine grained byte by byte.
249 * Preserve the compressed time on decompression where suitable.
250 * Store a copy of the control struct to be reused on subsequent files to prevent
251 variables being modified in the control struct on the first file that corrupt
252 compression/decompression of the 2nd file.
253 * Explicitly select C99 to avoid certain warnings.
254 * Generic modifications to silence -Wextra warnings.
255 * Fix typos.
256 * Use an array of parameters in lrztar to allow working with directories with
257 spaces in their names.
260 SEPTEMBER 2011, version 0.608 Con Kolivas
261 * get_sb only allows accessing one byte at a time, yet we don't need that
262 functionality when sliding mmap is not in use. Use different versions of the
263 function and the larger memcpys depending on whether sliding mmap is in use or
264 not. This affords a substantial speedup in the rzip phase of files not requiring
265 sliding mmap.
266 * A small optimisation of moving the check for remapping the low buffer also
267 speeds up the sliding mmap version slightly.
268 * Installation of the included bash completion script was disabled to avoid
269 conflict with distribution bash completion packages.
270 * Minor cleanup of sha4.h
271 * Make mremap use the fake mremap on all platforms not linux.
273 SEPTEMBER 2011, version 0.607 Con Kolivas
274 * Updated lzma library to version 920.
275 * Fixed a rare unable-to-decompress corner case.
276 * Added lrzip completion script.
277 * Updated makefile to more portable posix version.
278 * Hopefully fixed compilation on FreeBSD not supporting memopen.
279 * Added lots more debugging information about offsets in max verbose mode.
280 * Removed error and warning messages that are spammy or harmless.
282 MAY 2011, version 0.606 Con Kolivas
283 * lrzuntar broke as lrzip doesn't automatically use stdout now, so use lrzcat
284 in the lrztar script instead.
286 MAY 2011, version 0.605 Con Kolivas
287 * Addition of explicit lrzcat symlink which automatically sets "-d -o -"
288 * Revert automatic output to stdout due to addition of explicit lrzcat
289 executable, and to be consistent with gzip.
290 * Fix counter from being zeroed on every loop in rzip.c which was creating
291 excess progress output.
292 * Update GNU license.
293 * Set aside at least page_size of zeroes when compressing a chunk, thus
294 allowing files that are otherwise too small to compress with lrzip to be
295 compressed. The zeroes will just be dropped on decompression.
296 * Add explicit checks to cope with zero sized files as well.
297 * Remove the arbitrary 128 byte minimum size restriction on archives now that
298 any size file can be safely archived.
299 * Set the eof flag when reading older file format files to not hang on zero
300 byte sized archives.
301 * Update the percentage counter when getting file info to not overflow with
302 tiny files.
303 * Set the maximum file mode to 0666 to avoid setting the executable bit when
304 permissions are not carried over (such as from STDIO).
306 APRIL 2011, version 0.604 Con Kolivas
307 * Detach threads after creating them on the compression side. Not joining them
308 meant that compressing massive files requiring hundreds of threads would
309 eventually hit the resource limit of number of threads created even though
310 the threads themselves would exit.
312 APRIL 2011, version 0.603 Con Kolivas, George Makrydakis, Jari Aalto.
313 * lseek in stream.c wasn't being compiled to the lseek64 variant on Apple
314 due to missing includes, breaking >2GB files. Added includes.
315 * Detect when stdout is being redirected and automatically direct output to
316 stdout unless a filename is specified.
317 * Update lrztar to properly support -S -O and -o, and use new syntax not
318 requiring '-o -' for stdout.
319 * Update lrzip.conf to support encryption.
320 * Do a sanity check to ensure lrzip is not attempting to work on a directory.
321 * Typo fixes.
323 APRIL 2011, version 0.602 Con Kolivas
324 * Fixed the symlinks breaking package generation.
325 * Made maximum chunk allocable on 32bits 2/3 of a GB again limiting total ram
326 used to 2GB, regardless of how much crazy ram people have put on such systems
327 since the rest is unusable.
328 * Now builds on Cygwin courtesy of Тулебаев Салават.
329 * Check for podman on ./configure.
330 * Disable md5 generation and checking on Apple till it's fixed.
331 * Display full version when -V is invoked.
332 * Don't restrict file permissions from unknown source, respecting umode
335 MARCH 2011, version 0.601 Con Kolivas
336 * Fixed the symlinks for lrzuntar, lrunzip not being installed.
337 * Fixed the pod-based manpages not being installed.
338 * Clearly demonstrate that ASM is not supported on 64bits.
340 MARCH 2011, version 0.600 Con Kolivas
341 * Massive rewrite with new file format to accomodate new features.
342 * Allocate temporary buffers of safely sized ram that can act as temporary
343 storage for performing de/compression to/from STDIN/STDOUT without requiring
344 temporary physical files. Files compressed on machines with much larger ram
345 being decompressed on smaller ram machines may still require temporary files,
346 but as much as possible is done using in-ram decompression, and minimally
347 sized temporary files. Information displayed is more verbose and accurate in
348 STDIN/STDOUT mode.
349 * The temporary buffers created for decompressing to STDOUT are also used
350 for decompressing regular files now avoiding multiple write/reads when
351 re-constructing the file on decompression. This can dramatically speed up
352 the rzip phase of decompression on complicated files with many small matches,
353 and always makes testing faster.
354 * Compress block headers as well now since we know how many bytes can be used
355 to describe the length of the block, decreasing overall file size.
356 * Store the rzip chunk size per chunk to make it possible to check total file
357 size by summating each rzip chunk size when it's not known till the end (as
358 happens when compressing from STDIN).
359 * Implement password protected encryption. Import the polarssl code for
360 sha512 and aes128 routines. Read password without echoing to screen by
361 disabling echo via terminfo. Take the password then multiply hash it
362 according to the date it was generated. Inrease the number of hashes according
363 to Moore's law so it always takes approximately 1 second per password on the
364 most modern hardware when first encrypted. Hash the password against 8 bytes
365 of salt which is a combination of the 2 byte encoded loop counter (for how
366 many times to hash the password) and 6 random bytes. Take random from
367 /dev/urandom if it's available and fall back to random() if not. Encrypt each
368 block of compressed data with 8 extra bytes of random salt. Once the headers
369 are written, go back and encrypt the headers as well. Then encrypt the md5
370 hash value as well. Anything beyond the initial lrzip magic header should
371 apppear as random data and no two successive encryptions of the same data with
372 the same password should generate the same data.
373 * New build system should be more robust and portable.
374 * Abstract out functions better into separate files and headers, and remove
375 all use of global variables. This will make the generation of an lrzip
376 library possible in the future.
377 * Prevent testmalloc from coming up with a negative number when determining
378 how big a block of memory to allocate by decreasing the number of threads to
379 be used and then aborting to a minimum value should it still be too much
380 apparent ram.
381 * Big Endian fixes.
382 * Change lrztar and lrzuntar to not use temporary files.
383 * Countless other fixes, documentation and cleanups.
385 MARCH 2011, version 0.571 Con Kolivas
386 * Only retry mmaping if it's a memory error, otherwise it may give spurious
388 * Check for free space before compression/decompression and abort if there
389 is inadequate free space if the -f option is not passed.
390 * Fix the wrong check in rzip.c which was rounding down the page size and
391 making for one extra small chunk at the end.
392 * Check the correct stdout when refusing to pipe to a terminal.
393 * Fix windows EOL on lzma.txt.
394 * Ignore what stdout is going to when testing from stdin.
395 * More verbose summary after we know whether we have stdin/out to more
396 accurately reflect the window that will be used.
397 * Updated gitignore
398 * Unlink temporary files immediately to avoid files lying around.
399 * Check free space AFTER reading magic, and not when decompressing to stdout.
400 * Don't dump output to stdout when just testing a file.
401 * Dump the temporary file generated on emulating stdout on decompression after
402 every chunk is decompressed instead of after the whole file is decompressed.
403 * Don't attempt to file check decompression when it's not meaningful (test
404 only or to stdout).
405 * Clean up and simplify the times displayed component. It could potentially
406 show thousands of seconds.
407 * Remove open coded strerror messages when they're to be printed by fatal().
409 MARCH 2011, Michael Blumenkrantz
410 * Updated autotools/conf build system.
412 FEBRUARY 2011, version 0.570 Con Kolivas
413 * Change the lzo testing to a bool on/off instead of taking a parameter.
414 * Clean up the messy help output.
415 * Refuse to read from or write to terminal in stdin/stdout mode.
416 * Delete temporary files generated when testing from stdin.
417 * Don't call perror on failures that aren't system related.
418 * Improve visual output.
419 * Significant multi-threading speed-ups - as the spawning of threads is
420 partially serialised, use one more thread than CPUs.
421 * Don't split up the chunks into multiple threads when no back end
422 compression is being used.
423 * Take into account the precise amount of overhead that the memory hungry
424 back-end compression modes lzma and zpaq use when determining how much ram to
425 allocate, and limit total back-end compression ram to 1/3.
426 * Deprecate the -M option as it is no longer of any significant utility
427 compared to the -U option.
428 * Strip symbols by default in compilation and add -W -Wall to CXXFLAGS to be
429 consistent with CFLAGS.
430 * Update benchmarks.
431 * Add verbose and max verbose modes for -i information giving a breakdown of
432 each rzip chunk and compressed block.
433 * Modify purpose of -T threshold option to mean Disable LZO pre-compression
434 testing. Threshold level test deprecated.
435 * Make failure to allocate a buffer in the compress_ functions put out an
436 error message always rather than just in max_verbose mode.
438 FEBRUARY 2011, Peter Hyman
439 * Minor updates to man pages, lrzip.conf example file.
440 * Update main help screen to include environment settings.
441 * Update to respect $TMP environment variable for TMP files.
442 * Updated control structure to include tmpdir pointer.
443 * Update lrzip.conf parser to respect -U, -H, -T, k, and -c options.
444 * Update lrzip.conf example to include new parameters.
445 * Reorder main Switch loop in main.c for readability.
446 * Have UNLIMITED and control.window be exclusive. UNLIMITED wins.
447 * Corrects heuristic computation in rzip.c which would override
448 UNLIMITED if control.window set
449 * Show heuristically computed control.window when computed.
450 * Remove display compression level from control.window verbose output.
451 * Update print_verbose format for Testing for incompressible data in stream.c
452 to omit extra \n.
454 FEBRUARY 2011, version 0.560 Con Kolivas
455 * Rewrote the locking implementation to use pthread_mutexes instead of
456 semaphores. This makes multi-threading work on OSX.
457 * Re-use a single malloc in unzip_match instead of multiple malloc/frees.
458 * Imported the md5 code from coreutils and modified them to be used by lrzip.
459 * Implemented md5 hash checking on compression and decompression by using
460 buffers to pass to the existing crc checking and re-using them for the md5
462 * Sped up the hash checking on compression by not reading the whole chunk at
463 once to do the hash check, and instead read one paged size chunk.
464 * Store the md5 hash value at the end of the archive, maintaining backward
465 compatibility with existing 0.5 versions of lrzip, and place a flag in the
466 header denoting its existence.
467 * Use the header flag for md5's existence to determine what hash check to
468 perform on decompression.
469 * Add hash value display to maximum verbose output and in information mode.
470 * Add -H option to enable hash value displaying without verbose mode.
471 * Display what hash checking will be used on decompression in verbose mode and
472 in information displayed with -i.
473 * Add -c option to perform a hash check on the file generated on decompression,
474 comparing it to that stored in the lrzip archive, or that generated on
475 decompression on archives that do not have it stored.
476 * Delete broken or damaged files generated on interruption or when they fail
477 integrity testing by default.
478 * Implement the -k option to keep broken or damaged files.
480 FEBRUARY 2011, version 0.552 Con Kolivas
481 * Fix incompressible blocks causing failure with bzip2 and gzip.
482 * Freebsd doesn't do mremap, use the fake one.
483 * Update copyright notices.
484 * Flush writes to disk before allocating ram.
485 * Relax memory allocation when no back end compression will be used.
486 * Clean up unnecessary semaphore usage in decompression.
487 * Fix minor typos.
488 * Convert mmaps to malloc in runzip as they may silently fail if not a
489 multiple of page size, thereby leading to silent corruption on decompression.
490 * Return the correct length by unzip_literal.
491 * Use wrappers to serialise threads on OSX and bypass use of unnamed semaphores
492 which aren't supported.
494 DECEMBER 2010, version 0.551 Con Kolivas
495 * Fix stdin failing due to getting sizes all wrong.
496 * Fix stdin compression values not being shown at end.
497 * Fix inappropriate failure when lzma doesn't compress block.
499 DECEMBER 2010, version 0.550 Con Kolivas
500 * Move the threading on compression to higher up in the code, allowing the next
501 stream to start using compression threads before the previous stream has
502 finished. This speeds up compression on files that take more than one pass to
504 * Limit the number of threads decompressing stream 0 to just 1 since it's always
505 followed by stream 1 chunks, and it may lead to failure to decompress due to
506 running out of memory by running too many threads.
507 * Default compression level and window size on lzma is set to 7 which is the
508 highest it goes. Scale the 9 lrzip levels into 7, thus making the default lzma
509 level 5 which uses a lot less memory and is substantially faster at the cost of
510 some compression.
511 * Rationalise the memory testing now that the default lzma settings use a lot
512 less ram by default, and make all systems use no more than 1/3 ram in one mmap.
513 This allows larger windows to be used by 32 bit at last without memory
514 allocation errors.
515 * Revert "Make threads spawn at regular intervals along chunk size thus speeding
516 up compression" as it actually slowed it down instead of speeding it up.
517 * Cope with compression/decompression threads failing by waiting till the
518 previous thread has finished its work, thus serialising the work and using less
519 ram, making success more likely.
520 * Fix some dodgy callocs which weren't really allocating enough ram.
521 * Destroy semaphores used in stream_in on closing the stream.
522 * Minor output improvements.
524 DECEMBER 2010, version 0.544 Con Kolivas
525 * Make multiple stream 0 entry decompression more robust by creating separate
526 thread groups for stream 0 and stream 1.
527 * uclibc fails to return valid values for memory size with sysconf so create
528 a workaround.
529 * 32 bits has memory allocation errors on large files so be ruthless with
530 limiting window sizes and use more sliding mmap.
531 * Split up compression streams according to proportion of hash search thus
532 evenly distributing work to threads based on raw data length rather than
533 rzip pre-processed data. This makes for more CPU usage on compression and
534 smoother write out on decompression with small speed ups.
535 * Remove unnecessary argument passing.
536 * Fix -U causing a floating point error with ultra small sized files.
537 * 32 bits still sometimes fails the lzma compression stage due to internal
538 memory allocation issues, so use bzip2 as a fallback on blocks that fail to
540 * Minor cleanups.
542 NOVEMBER 2010, version 0.543 Con Kolivas
543 * Fix for when there are multiple stream 0 entries per chunk, which would
544 cause a failure on decompression.
545 * Fix lzma windows being set far too small by mistake at all times.
546 * Decrease nice level of main process compared to back end threads as it tends
547 to be the rate limiting step.
549 NOVEMBER 2010, version 0.542 Con Kolivas
550 * Choose sane defaults for memory usage since linux ludicriously overcommits.
551 * Use sliding mmap for any compression windows greater than 2/3 ram.
552 * Consolidate and simplify testing of allocatable ram.
553 * Minor tweaks to output.
554 * Round up the size of the high buffer in sliding mmap to one page.
555 * Squeeze a little more out of 32 bit compression windows.
556 * Fix sliding mmap from not working on 2nd pass onwards.
558 NOVEMBER 2010, version 0.541 Con Kolivas
559 * Fix wrong number of passes reported.
560 * Re-fix the off-by-one that wasn't off-by-one.
561 * Limit lzma compression windows to 300MB as per reports of failures with larger
564 NOVEMBER 2010, version 0.540 Con Kolivas
565 * Massive rewrite of backend decompression phse, implementing multithreading.
566 This is done by taking each stream of data on read in into separate buffers for
567 up to as many threads as CPUs. As each thread's data becomes available, feed it
568 into runzip once it is requests more of the stream. Provided there are enough
569 chunks in the originally compressed data, this provides a massive speedup
570 potentially proportional to the number of CPUs. The slower the backend
571 compression, the better the speed up (i.e. zpaq is the best sped up).
572 * Fix the output of zpaq compress and decompress from trampling on itself and
573 racing and consuming a lot of CPU time printing to the console.
574 * When limiting cwindow to 6 on 32 bits, ensure that control.window is also set.
575 * When testing for the maximum size of testmalloc, the multiple used was out by
576 one, so increase it.
577 * Minor output tweaks.
578 * Build warning fixes.
579 * Updated benchmarks.
581 NOVEMBER 2010, version 0.530 Con Kolivas
582 * Massive rewrite of backend compression phase. Now the stream is split up
583 into as many chunks as there are CPUs, of at least 10MB in size, that are
584 still mallocable. Once the stream has reached a chunk of this size, its buffer
585 is handed to a new backend compression thread which works while the rzip stream
586 continues processing. This has the effect of parallelising workloads almost
587 linearly up to the number of CPUs on the slower compression backends. ZPAQ,
588 in particular, is effectively 4x faster on quad core now. Decompression is
590 * Added the -p option to allow the number of processors to be specified to
591 override the detected number.
592 * Changed the default level back to 7 as 9 wasn't offering significanly more
593 compression but was adding time.
594 * Increased the size of all the buffers to other backends now as well, since
595 each block adds overhead with its header.
596 * Numerous alterations to screen output to cope with new threaded compression
598 * Deprecated the -P option since not setting the file permissions only
599 generates a warning now, not a failure.
600 * Updated docs and benchmarks.
602 NOVEMBER 2010, version 0.520 Con Kolivas
603 * Distros don't like 3 point version numbering so just repackaged as 0.520.
605 NOVEMBER 2010, version 0.5.2 Con Kolivas
606 * Fixed the Darwin build, again.
607 * Fixed cases of extreme ram usage on 32 bit failing by limiting zpaq to 600MB
608 windows as well.
609 * Check page size if we can instead of assuming it's always 4k.
610 * Improve the progress output.
611 * Change failure to chmod and failure to set nice level to warnings only.
612 * Standardise what's a stderr message and what's output.
614 NOVEMBER 2010, version 0.5.1 Con Kolivas
615 * Fix Darwin build - Darwin doesn't support mremap so introduce a fake wrapper
616 for it.
617 * Fix the memopen routines, a wrongly implemented wrapper for Darwin equivalents
618 was also using the faked versions on all builds.
619 * Fix dodgy ordered includes.
620 * Clean up excessive use of #ifdefs
621 * Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of
622 accessing the buffer directly, thus allowing us to have window sizes larger than
623 available ram. This is implemented through the use of a "sliding mmap"
624 implementation. Sliding mmap uses two mmapped buffers, one large one as
625 previously, and one smaller one. When an attempt is made to read beyond the end
626 of the large buffer, the small buffer is remapped to the file area that's being
627 accessed, while the larger one is remapped as the search progresses along the
628 file. While this implementation is potentially much slower than direct mmapping,
629 it allows us to implement unlimited sized compression windows.
630 * Implement the -U option with unlimited sized compression windows.
631 * Rework the selection of compression windows. Instead of trying to guess how
632 much ram the machine might be able to access, we try to safely buffer as much
633 ram as we can, and then use that to determine the file buffer size. Do not
634 choose an arbitrary upper window limit unless -w is specified.
635 * Rework the -M option to try to buffer the entire file, reducing the buffer
636 size until we succeed.
637 * Align buffer sizes to page size.
638 * Clean up lots of unneeded variables.
639 * Fix lots of minor logic issues to do with window sizes accepted/passed to rzip
640 and the compression backends.
641 * More error handling.
642 * Change -L to affect rzip compression level directly as well as backend
643 compression level and use 9 by default now.
644 * Fix file size reporting on compressed files generated from stdin.
645 * More cleanups of information output and more information.
646 * Add chunk percentage to progress update.
647 * Reinstated the 2GB buffer limit on 32 bit machines during compression, though
648 the -U mode can work around it now.
649 * Code micro-optimisations.
650 * Use 3 point release numbering in case one minor version has many subversions.
651 * Numerous minor cleanups and tidying.
652 * Updated docs, manpages, and benchmarks.
654 NOVEMBER 2010, version 0.5 Con Kolivas
655 * Changed offset encoding in rzip stage to use variable byte width offsets
656 instead of 64 bits wide. Makes for better compression and slightly faster.
657 * Write the byte width into the file before each block.
658 * Shrunk match lengths to maximum of 16 bits again as per original rzip as the
659 larger offsets did not achieve greater compression and made final size larger.
660 * New file format not backwards compatible due to variable byte widths.
661 * Rewrote memory initialisation to have a pre-allocation stage to try and
662 find the maximum memory usable and defragment ram.
663 * Use reduced window size if allocating memory fails at higher size.
664 * Change use of malloc to mmap to make it possible to address up to 44 bit
665 sized offsets even on 32 bit machines on decompression. Still unable to use
666 greater than 2GB windows on 32 bit machines and unsure if this is fixable.
667 * Reworked the STDIN code to use an anonymous mmap and read in stdin into this
668 to make it possible to compress from STDIN without the need for temporary
669 files. As the file size is not known in advance, memory allocation is set to
670 large and byte width to equivalent size.
671 * Reallocation of ram where possible to minimise risk of running out of memory
672 in the middle of a compression phase, and flushing to disk to empty dirty ram
673 for the same reason.
674 * More robust fatal warnings.
675 * Numerous cleanups and tidying of code and addition of comments.
676 * Updated documentation to reflect changes.
678 OCTOBER 2010, version 0.47, Con Kolivas
679 * Fix the symlinking problem when DESTDIR is in use reported by a billion
682 MAY 2010, version 0.46, Con Kolivas, Ed Avis.
683 * Suppress final [OK] message with -q flag EA
684 * Handle mkstemp() errors correctly EA
685 * Add lrzuntar manpage
686 * Update manpages
688 APRIL 2010, version 0.45, Con Kolivas, Jon Tibble, George Makrydakis
689 * Fixes the nasm program test (AC_CHECK_PROG doesn't overwrite a
690 variable that is already set so do it manually) JT
691 * Fix compiler flags as not all compilers accept -Wall -W (cc on
692 Solaris/OpenSolaris) JT
693 * Fix lrztar to not try to compress files already with the .lrz extension GM
694 * Fix lrztar to decompress files where the pathname is ../* GM
695 * Add lrzuntar symlink to call lrztar -d
698 MAR 2010, version 0.45, Con Kolivas, Jari Aalto
699 * Fixed reported window size
700 * Fixed 32bit windows being attempted to be larger than contiguous amounts
701 by taking into account VM kernel/userspace split of 896MB.
702 * Minor code cleanups
703 * Added lrztar and lrunzip docs
704 * Fix minor typos
705 * Added distclean and maintainer-clean make targets
708 DEC 2009, version 0.44, Con Kolivas, George Makrydakis
709 * Added lrztar wrapper to manage whole directories.
710 * Added -i option to provide information about a compressed file.
711 * Fixed "nan" showing as Compression speed on very small files.
712 * Fixed build for old bz library.
713 * Avoid overwriting output file if input doesn't exist.
714 * Implement signal handler to delete temporary files.
717 DEC 2009, version 0.43, Con Kolivas, Jukka Laurila
718 * Darwin support thanks to Jukka Laurila.
719 * Finally added stdin/stdout support due to popular demand. This is done
720 by basically using temporary files so is a low performance way of using
722 * Added test function. This just uses a temporary file during decompression.
723 * Config files should now accept zpaq options.
724 * Minor code style cleanups.
725 * Updated benchmarks in docs.
726 * Add a warning when attempting to decompress a file from a newer lrzip
730 NOV 2009, version 0.42, Con Kolivas
731 * Changed progress update to show which of 2 chunks are being compressed
732 in zpaq.
733 * Fixed progress update in ZPAQ to not update with each byte which was
734 wasting heaps of CPU time.
737 NOV 2009, version 0.41, Con Kolivas
738 * Added zpaq compression backend for extremely good but extremely slow
739 compression (incompatible with previous versions if used).
740 * Limited chunk size passed to LZMA to 4GB to avoid library overflows.
741 * Minor changes to the formatting output
742 * Changed lower limit of -T threshhold to 0 to allow disabling it.
743 * Added lzo_compresses check into zpaq and bzip2 as well since they're
747 NOV 2009, version 0.40, Con Kolivas
748 * Massive core code rewrite.
749 * All code moved to be 64bit based for compression block addressing and length
750 allowing compression windows to be limited by ram only.
751 * 64bit userspace should now have no restriction on compression window size,
752 32bit is still limited to 2GB windows due to userspace limitations.
753 * New file format using the new addressing and data types, incompatible with
754 versions prior to 0.40.
755 * Support for reading and decompressing older formats.
756 * Minor speedups in read/write routines.
757 * Countless minor code fixes throughout.
758 * Code style cleanups and consistency changes in core code.
759 * Configure script improvements.
762 NOV 2009, version 0.31, Con Kolivas
763 * Updated to be in sync with lzma SDK 9.07beta.
764 * Cleanups and fixes of the configure scripts to use the correct package version
766 * Massive fixes to the memory management code fixing lots of 32bit overflow
767 errors. The window size limit is now 2GB on both 32bit and 64bit. While it
768 appears to be smaller than the old windows, only 900MB was being used on .30
769 even though it claimed to use more. This can cause huge improvements in the
770 compression of very large files.
771 * The offset when mmap()ing was not being set to a multiple of page size so
772 it would fail if the window size was not a multiple of it.
773 * Flushing of data to disk between compression windows was implemented to
774 minimise disk thrashing of read vs write.
777 NOV 2009, version 0.30, Con Kolivas
778 * Numerous bugfixes to try and make the most of 64bit environments with huge
779 memory and to barf less on 32bit environments.
780 * Executable stacks were fixed.
781 * Probably other weird and wonderful bugs have been introduced.
782 * -P option to not set permissions on output files allowing you to write to
783 braindead filesystems (eg fat32).
786 JAN 2009, version 0.24, Peter Hyman, email@example.com
787 Happy New Year!
788 * Upgrade LZMA SDK to 4.63. Use new C Wrapper. Invalidates
789 LZMA archives created earlier due to new Magic property
791 * New LZMA logic will automatically determine allow LZMA
792 code to determine optimal lc, lp, pb, fb, and dictionary
793 size settings. stream.c will only pass level and thread
794 information. Compress function will return encoded 5 byte
795 data with compression settings. This will be stored in lrz
796 file header.
797 * add error messages during LZMA compression. There are some
798 edge cases where LZMA cannot allocate memory. These errors
799 are reported and the user will be advised to use a lower
800 compression window setting.
801 * type changes in rzip_fd function for correctness.
802 * remove function *Realloc() since it was never used. Cleaned
803 in rzip.h and util.c.
804 * apply munmap prior to closing and compressing stream in
805 function rzip_chunk in rzip.c.
806 * add realloc function in close_stream_out in stream.c
807 to reclaim some ram and try and allieviate out of memory
808 conditions in LZMA compression.
809 * remove file acconfig.h and include DEFINE in configure.in.
810 * add lrzip.conf capability.
811 * add timer for compression including elapsed time and eta.
812 * add compression and decompression MB/s calculation.
813 * Updated WHATS-NEW, TODO and created BUGS file.
814 * Updated lrzip.1 manpage and created lrzip.conf.5 manpage.
815 * Added lrzip.conf.example file in doc directory.
817 MAR 2008, Con Kolivas, firstname.lastname@example.org
818 * Numerous changes all over to place restrictions on window
819 size to work with 32 bit limitations.
820 * Various bugfixes with respect to detecting buffer sizes and
821 likelihood of compressibility.
822 * Fixed the inappropriate straight copying uncompressed data for
823 files larger than 4GB.
824 * Re-initiated the 10MB window limits for non-lzma compression.
825 I was unable to reproduce any file size savings.
826 * Allow compression windows larger than ramsize if people really
827 really want them.
828 * Decrease thresholds for the test function to a minimum of 5%
829 compressibility since the hanging in lzma compression bug has been
832 JAN 2008, version 0.22, Peter Hyman, email@example.com
833 * version update
835 Thanks to Lasse Collin for debugging the problem LZMA
836 had with hanging on uncompressable files.
837 Update for control parameters to both compress and
838 decompress functions.
840 * use of @top_srcdir@ (Lasse Collin). Also moved away
841 more cruft.
842 main.c stream.c.rzip.h LZMALib.cpp lzmalib.h
843 * addition of three new control structure members.
844 control.lc -- literal context bits
845 control.lp -- literal post state bits
846 control.pb -- post state bits
847 These are needed to ensure decompression will work.
848 These will now be stored along with control.compression_level
849 in the lrz file beginning at offset 0x16 for three bytes.
850 These will be passed to the functions lzma_compresses and
851 lzma_uncompress. Currently, only compression level is
852 needed or used, but the others are stored for possible future
854 See magic file for more information.
856 * Change to lzo_compresses function that will reject a chunk
857 without testing it if the size of the chunk is greater
858 than the compression window * threshold. This is to avoid
859 a low probability that lzma would still be passed a chunk
860 that contains uncompressible data or barely compressible
861 data. If after rzip hashing the chunk size is still close
862 to the window size, there is hardly anything worth
863 compressing. While there is no reason lzma cannot get the
864 chunk, this will save a lot of time.
866 * updated file to show new layout that includes lzma
869 * added warning about using lrzip-0.22 with earlier versions.
871 * highlight of new features.
873 DEC 2007, version 0.21. Peter Hyman, firstname.lastname@example.org
874 * version update.
875 * Modified to use Assembler routines from lzma SDK for CRC
876 computation when hashing streams in rzip.c and runzip.c.
877 Added files 7zCrcT8.c and 7zCrcT8u.s to lzma tree.
878 Cleaned up source tree. Moved unused files out of the way.
879 Moved non-core docs to doc directory
881 * correct AC_INIT to set program variables.
882 * modified to add check for nasm assembler.
883 * modified syntax of test for errno in error.h to use
884 echo $ECHO_N/$ECHO_C instead of $ac_n/$ac_c which
885 was incorrect.
886 Makefile.in, lzma/Makefile
887 * modified to add compile instructions for 7zCrcT8.c
888 and 7zCrcT8U.s and Assembler. Cleaned up to remove
889 targets that don't exist or sources that don't
891 Modified to properly set directories. Added doc install.
892 Add link command to symlink lrunzip to lrzip.
894 Add CrcGenerateTable() function to init CRC tables.
895 This is needed for all crc routines including those
896 in MatchFinderMT.
897 rzip.c and runzip.c
898 * Updated source to change call to crc32_buffer to call
899 CrcUpdate in the assembler code. Changed parameter order
900 to conform.
902 * Removed 10MB limit on streams for bzip, gzip, and lzo.
903 This, to improve effeciency of long range analysis. For
904 some files, this could improve results.
906 * Added file to keep benchmarks current to version.
907 (probably need to update README too).
909 * Explain how to remove default compile of Assembler
911 config.sub config.guess
912 * added files for system detection.
914 DEC 2007, version 0.20. Peter Hyman, email@example.com
916 * Updated to LZMA SDK 4.57.
917 * Updated to p7zip POSIX version. (www.p7zip.org)
918 * Added multi-threading support (up to 2x speed with LZMA).
919 * Edited LZMADecompress.cpp for backward compatibility
920 with decompress function. Needed SetPropertiesRaw function.
921 * Repopulated source tree for distribution.
922 * Updated Makefile.in to reflect new source files.
923 Updated to include command to link lrunzip to lrzip because
924 lrzip will test if lrunzip was used on command line.
925 * Updated Makefile.in for new compile time and linking options.
926 * Updated LZMALibs.cpp to include new property members for
927 LZMAEncoders as well as changed default dictionaries to
928 level+16. This would make the default compression level
929 of 7 translate to a dictionary number of 23.
930 * Added output to show Nice Level when verbose mode set
931 Initial add of support for zlib which seems to give quite
932 excellent performance.
933 * configure.in added AC_CHECK for libz and libm.
934 Added AC_PROG_LN_S for Makefile symlink section.
935 * lrzip.1 updated man page for -g option
936 * main.c added option test for gzip
937 Added sysconf(_SC_NPROCESSORS_CONF) for CPU detection
938 for threading.
939 Updated verbose output to show whether or not
940 Threading will be used.
941 Added Timer for each file compressed.
942 * rzip.h added flags for GZIP compression.
943 Added control member for threads. Arg passed to
945 * stream.c update to accomodate gzip compress and decompress
946 functions. Cleaned up file by rearranging functions into
948 Removed include of lzmalib.h since it was causing a
949 compile time warning with zlib.h. Prototyped functions
951 Cleanup output from lzo_compresses function so that
952 unnecessary linefeeds are eliminated.
953 lzma_compress function call now uses threads as argument.
954 * Added README.benchmarks file to explain a method of
955 comparing results between different methods.
956 * LZMALib.cpp, lzmalib.h. Adjust function lzma_compress
957 prototype and function to include new argument threads.
958 This parameter is now placed in properties.
959 * lzma/Makefile. Updated to reflect new API library.
960 Updated to include Threading option.
962 DEC 2007, version 0.19. Con Kolivas.
963 * Added nice support, defaulting to nice 19.
965 DEC 2007, version 0.19. Peter Hyman, firstname.lastname@example.org
967 * Major goal was to stop LZMA from hanging on some files.
968 Accomplished this with a threhold setting that is used by
969 the lzo_compresses function to better analyze chunk data.
970 Threshold makes it less likely that uncompressible data
971 will be passed to the LZMA compressor.
974 * Added Threshold option 1-10 to control LZMA compression attempt.
975 Default value=2. This means that anything over 10% compression
976 as reported by lzo_compresses will return a true value to
977 the LZMA compression function.
978 * Added verbosity option and more verbosity option (-v[v]).
979 * Added -O option to specify output directory.
980 * Updated compress_file and decompress_file functions to handle.
981 output directories and better handle multi files and filename
982 extensions. Optimized some string handling routines.
983 Improved flexibility in determining location of output files
984 when using -O. Added fflush(stdout) to improve printf reliability.
985 * decompress_file will accept any filename and will automatically
986 append .lrz if not present. Won't automatically fail.
987 * Added logic to protect against conflicting options such as
988 -q and -v, -o and -O.
989 * Added printout to screen of options selected. Will display
990 only when -v or -vv used.
991 * Adjusted several printf statements to avoid compiler
992 warnings (use %ll for long long int types).
995 * Added decompression progress indicator.
996 Will show percent decompressed along with bytes decompressed
997 and total to be decompressed. Will show if -q option NOT used.
1000 * Version incremented to 0.19.
1001 * Added flag DEFINESs for verbosity and more verbosity.
1002 * Updated control struct to include output directory and
1003 threshold value. Removed verbosity member.
1006 * Minor changes to handle display when verbosity set. Changed
1007 number format in some printf statements to properly handle
1008 unsigned data.
1011 * major overhaul of lzo_compresses function to use a threshold
1012 value when testing a data chunk to see if it is suitable for
1013 LZMA compression. Optimized test loop to improve performance
1014 and reduce number of passes. Improved output reporting depending
1015 on verbosity setting.
1016 * Added print controls for verbosity option.
1017 * Corrected if statements that tested for error condition of
1018 some lzo functions that only return a true value regardless.
1021 * updated man page to show new options and explain -T threshold.
1024 * updated README to explain -T threshold option.
1027 * Added this file to help explain the theory behind the rewrite
1028 of the lzo_compresses function and how to use the -T option.
1031 * wish list and future enhancements.
1034 * added file.