This Python module provides an xopen function that works like Python’s built-in open function but also transparently deals with compressed files. xopen selects the most efficient method for reading or writing a compressed file.
Supported compression formats are:
.gz).bz2).xz).zst) (optional)Open a file for reading:
from xopen import xopen
with xopen("file.txt.gz") as f:
content = f.read()
Write to a file in binary mode, set the compression level and avoid using an external process:
from xopen import xopen
with xopen("file.txt.xz", mode="wb", threads=0, compresslevel=3) as f:
f.write(b"Hello")
The xopen module offers a single function named xopen with the following signature:
xopen( filename: str | bytes | os.PathLike, mode: Literal["r", "w", "a", "rt", "rb", "wt", "wb", "at", "ab"] = "r", compresslevel: Optional[int] = None, threads: Optional[int] = None, *, encoding: str = "utf-8", errors: Optional[str] = None, newline: Optional[str] = None, format: Optional[str] = None, ) -> IO
The function opens the file using a function suitable for the detected file format and returns an open file-like object.
When writing, the file format is chosen based on the file name extension: .gz, .bz2, .xz, .zst. This can be overriden with format. If the extension is not recognized, no compression is used.
When reading and a file name extension is available, the format is detected from the extension. When reading and no file name extension is available, the format is detected from the file signature <https://en.wikipedia.org/wiki/File_format#Magic_number>.
filename (str, bytes, or os.PathLike): Name of the file to open.
If set to "-", standard output (in mode "w") or standard input (in mode "r") is returned.
mode, encoding, errors, newline: These parameters have the same meaning as in Python’s built-in open function except that the default encoding is always UTF-8 instead of the preferred locale encoding. encoding, errors and newline are only used when opening a file in text mode.
compresslevel: The compression level for writing to gzip, xz and Zstandard files. If set to None, a default depending on the format is used: gzip: 1, xz: 6, Zstandard: 3.
This parameter is ignored for other compression formats.
format: Override the autodetection of the input or output format. Possible values are: "gz", "xz", "bz2", "zst".
threads: Set the number of additional threads spawned for compression or decompression. May be ignored if the backend does not support threads.
If threads is None (the default), as many threads as available CPU cores are used, but not more than four.
xopen tries to offload the (de)compression to other threads to free up the main Python thread for the application. This can either be done by using a subprocess to an external application or using a library that supports threads.
Set threads to 0 to force xopen to use only the main Python thread.
Opening of gzip files is delegated to one of these programs or libraries:
gzip)For xz files, a pipe to the xz program is used because it has built-in support for multithreaded compression.
For bz2 files, pbzip2 (parallel bzip2) is used.
xopen falls back to Python’s built-in functions (gzip.open, lzma.open, bz2.open) if none of the other methods can be used.
xopen writes gzip files in a reproducible manner.
Normally, gzip files contain a timestamp in the file header, which means that compressing the same data at different times results in different output files. xopen disables this for all of the supported gzip compression backends. For example, when using an external process, it sets the command-line option --no-name (same as -n).
Note that different gzip compression backends typically do not produce identical output, so reproducibility may no longer be given when the execution environment changes from one xopen() invocation to the next. This includes the CPU architecture as igzip adjusts its algorithm depending on it.
bzip2 and xz compression methods do not store timestamps in the file headers, so output from them is also reproducible.
Optional Zstandard supportFor reading and writing Zstandard (.zst) files, either the zstd command-line program or the Python zstandard package needs to be installed.
threads parameter to xopen() is None (the default) or any value greater than 0, xopen uses an external zstd process.zstd program is available) or if threads is 0, the zstandard package is used.To ensure that you get the correct zstandard version, you can specify the zstd extra for xopen, that is, install it using pip install xopen[zstd].
--long=31 files can now be opened without throwing errors.#154: Support for gzip levels has been made more consistent. Levels 0-9 are supported. Level 11 which was only available when the pigz backend was present is not supported anymore. Level 0, gzip format without compression, lead to crashes when the gzip application backend was used as this does not have a -0 flag. xopen() now defers to other backends in that case.
#152: xopen() now accepts file-like objects for its filename argument.
#146, #147, #148: Various refactors for better code size and readability:
- PipedCompressionReader/Writer are now combined _PipedCompressionProgram class.
- _PipedCompressionProgram is binary-only. For text reading and writing it is wrapped in an
io.TextIOWrapperin thexopen()function.- Classes that derive from PipedCompressionReader/Writer have been removed.
#148: xopen's classes, variables and functions pertaining to piped reading and writing are all made private by prefixing them with an underscore. These are not part of the API and may change between releases.
igzip is no longer used as a (de)compression backend as python-isal's threaded mode is a better choice in all use cases.isal.igzip_threaded module rather than piping to external programs in applicable cases. This makes reading and writing to gzip files using threads more efficient..zst) files. This requires that the Python zstandard package is installed or that the zstd command-line program is available.gzip --no-name (or -n) on the command line). This allows files to be written in a reproducible manner.xz process. Contributed by @fanninpm.seek() and tell() to the PipedCompressionReader classes (for Windows compatibility)xopen() has gained the parameters encoding, errors and newlines with the same meaning as in open(). Unlike built-in open(), though, encoding is UTF-8 by default..bz2 files if threads is greater than zero (contributed by @DriesSchaumont).igzip program (part of Intel ISA-L) is now used for reading and writing gzip-compressed files at compression levels 1-3, which results in a significant speedup.pigz to use only a single process. pigz cannot use multiple cores anyway when decompressing. By default, it would use extra I/O processes, which slightly reduces wall-clock time, but increases CPU time. Single-core decompression with pigz is still about twice as fast as regular gzip.threads=0 for specifying that no external pigz/gzip process should be used (then regular gzip.open() is used instead).pigz use at most four threads by default. This limit previously only applied when writing to a file. Contributed by @bernt-matthias.pigz subprocess. This is faster than using gzip.open.threads parameter (passed on to pigz)The name xopen was taken from the C function of the same name in the utils.h file that is part of BWA.
Some ideas were taken from the canopener project. If you also want to open S3 files, you may want to use that module instead.
@kyleabeauchamp contributed support for appending to files before this repository was created.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4