Repository navigation
Release 1.9.0 - #311
Merged
Merged
Release 1.9.0#311
Conversation
… versions of python
Fix reported memory errors in asan environment by using the debugging…
ci: re-enable PyPy wheel builds
Fix zlib headers for preset dictionary compression
Discard compressed input buffer on backward gzip seek
* update python versions * Update version of setup python * Check is resize size == 0 to prevent resizing the empty string object * Fix workflow file error * Revert " Check is resize size == 0 to prevent resizing the empty string object" This reverts commit 246a307. * Prevent empty string object from being resized * Use newer miniconda version * Pin python version for dynamic builds * Stop testing pypy versions for dynamic builds. Conda-forge does not support PyPy anymore: https://conda-forge.org/news/2024/08/14/sunsetting-pypy/
Update ISA-L to 2.32.1
Throw an overflow error on big headers
GzipReader_dealloc used members that were not initialized yet when the buffer protocol failed. igzip.decompress("x") crashed with a segmentation fault.
IgzipDecompressor stored zdict without increasing its reference count, but released it on deallocation. Creating a few decompressors with a dictionary crashed the interpreter.
The error path returned without releasing the lock, so the next call on the same object hung forever.
flush(Z_FINISH) freed the level buffer but kept the pointer. Level 0 accepts a second Z_FINISH flush, which freed the buffer again.
When a stream ends, ISA-L moves next_in back over the unused bytes in its bit buffer. If an earlier call read those bytes, next_in points before the current input, and unused_data got bytes from outside the buffer. Save the bit buffer after each call and take those bytes from it. Also clear unconsumed_tail at the end of the stream, as CPython does since gh-158169.
The buffersize check set ValueError without returning, and failed argument parsing went unnoticed because PyArg_ParseTupleAndKeywords returns 0, not a negative value. Both raised SystemError, for example for a large block_size or a non-integer compresslevel in igzip_threaded.open.
argparse and shutil are only needed by the command line interface, and multiprocessing was only used for cpu_count, which os.cpu_count replaces. Importing igzip_threaded takes 4.8 ms instead of 11.1 ms on Python 3.14.
GzipFile.readline checks for a closed file through two Python calls for each line. Checking fileobj directly makes line iteration, readlines(), and readline() 1.13 times faster for FASTQ data and 1.5 times faster for lines of a few bytes.
Raise compression errors of `igzip_threaded` writers with more than one thread instead of waiting forever
main() chose the level with `args.compresslevel or _COMPRESS_LEVEL_TRADEOFF`, so level 0 fell back to level 2. Set the default level in the argument parser instead.
The test checked byte 4, the first byte of MTIME. The flags are in byte 3.
With 8 KiB blocks, the last 32 KiB of the previous block is the whole block, so the tests could not tell whether the writer took the dictionary from the end or the start of it.
Use assert_writer_closed(), which the error tests already use. It also checks that the threads are no longer alive.
pytest.raises(Exception) also passes on the SystemError that Python raises when decompress() returns NULL without setting an exception.
isal_zlib.decompress() and igzip_lib.decompress() turn a bufsize of 0 into 1 and reject a negative bufsize, as zlib.decompress() does. No test used those values. Without the first check, a bufsize of 0 makes decompress() loop forever.
…_readable io.BufferedReader.writable() and io.BufferedWriter.readable() return False without asking the raw stream, so the tests could not fail.
isal_inflate decompresses up to about 64 KiB ahead into its own buffer, so it read the 100 to 300 byte inputs of these tests in full, and the input buffer was never used. Use 512 KiB of random data and multiply the offsets by 1,000.
No test checked that open() passes threads to the writer. Without it, the writer uses one thread, and the threaded tests still passed.
When IGzipFile.__init__ does not set self.compress, GzipFile compresses with zlib instead, and the igzip tests still passed.
The getter of _last_mtime returned None when the stored time was 0, so a gzip header with a modification time of 0 read as None. Keep a separate flag that is set when the first header is read.
Fix `-0` and `--fast` in the `igzip` command line interface
Check the flags byte for `FNAME` in `test_compress_infile_out_file_no_name`
Test threaded writes with blocks larger than 32 KiB
Check that `test_close_writer` stops the worker threads
Check the raw streams in `test_reader_not_writable` and `test_writer_not_readable`
Check the exception type in `test_failure` of `IgzipDecompressor`
The uncompressed data is random bytes, not text, so RANDOM_TEXT becomes RANDOM_DATA and the compressed data becomes COMPRESSED_RANDOM_DATA.
Make the input buffer tests of `IgzipDecompressor` reach the input buffer
Test `bufsize` values of 0 and -1 in `decompress()`
Test that `igzip_threaded.open()` starts the given number of threads
Fix the `mtime` attribute of `IGzipFile` for a modification time of 0
The review asked for a test of the parameters instead of a byte-for-byte comparison of the deflate data. The test now records the calls to isal_zlib.compressobj and checks that f.compress is an isal_zlib.Compress.
Test that `IGzipFile` compresses with `isal_zlib` at the given level
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checklist