A RetroSearch Logo

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Search Query:

Showing content from https://en.wikipedia.org/wiki/XZ_Utils below:

XZ Utils - Wikipedia

From Wikipedia, the free encyclopedia

Lossless data compression software

XZ Utils (previously LZMA Utils) is a set of free software command-line lossless data compressors, including the programs lzma and xz, for Unix-like operating systems and, from version 5.0 onwards, Microsoft Windows. For compression/decompression the Lempel–Ziv–Markov chain algorithm (LZMA) is used. XZ Utils started as a Unix port of Igor Pavlov's LZMA-SDK that has been adapted to fit seamlessly into Unix environments and their usual structure and behavior.

XZ Utils can compress and decompress the xz and lzma file formats. Since the LZMA format has been considered legacy,[2] XZ Utils by default compresses to xz. In addition, decompression of the .lz format used by lzip is supported since version 5.3.4.[3]

In most cases, xz achieves higher compression rates than alternatives like zip,[4] gzip and bzip2. Decompression speed is higher than bzip2, but lower than gzip. Compression can be much slower than gzip, and is slower than bzip2 for high levels of compression, and is most useful when a compressed file will be used many times.[5][6]

XZ Utils consists of two major components:

Various command shortcuts exist, such as lzma (for xz --format=lzma), unxz (for xz --decompress; analogous to gunzip) and xzcat (for unxz --stdout; analogous to zcat).

Both the behavior of the software and the properties of the file format have been designed to work similarly to those of the popular Unix compressing tools gzip and bzip2.

Just like gzip and bzip, xz and lzma can only compress single files (or data streams) as input. They cannot bundle multiple files into a single archive – to do this an archiving program is used first, such as tar.

Compressing an archive:

xz   my_archive.tar    # results in my_archive.tar.xz
lzma my_archive.tar    # results in my_archive.tar.lzma

Decompressing the archive:

unxz    my_archive.tar.xz      # results in my_archive.tar
unlzma  my_archive.tar.lzma    # results in my_archive.tar

Version 1.22 or greater of the GNU implementation of tar has transparent support for tarballs compressed with lzma and xz, using the switches --xz or -J for xz compression, and --lzma for LZMA compression.

Creating an archive and compressing it:

tar -c --xz   -f my_archive.tar.xz   /some_directory    # results in my_archive.tar.xz
tar -c --lzma -f my_archive.tar.lzma /some_directory    # results in my_archive.tar.lzma

Decompressing the archive and extracting its contents:

tar -x --xz   -f my_archive.tar.xz      # results in /some_directory
tar -x --lzma -f my_archive.tar.lzma    # results in /some_directory

Single-letter tar example for archive with compress and decompress with extract using short suffix:

tar cJf keep.txz keep   # archive then compress the directory ./keep/ into the file ./keep.txz
tar xJf keep.txz        # decompress then extract the file ./keep.txz creating the directory ./keep/

xz has supported multi-threaded compression (with the -T flag)[7] since 2014, version 5.2.0;[3] since version 5.4.0 threaded decompression has been implemented. Threaded decompression requires multiple compressed blocks within a stream which are created by the threaded compression interface. The number of threads can be less than defined if the file is not big enough for threading with the given settings or if using more threads would exceed the memory usage limit.[7]

An xz file is a sequence of one or more streams. There may be null bytes (padding) after each stream.

The xz format improves on lzma by allowing for preprocessing filters (BCJ and delta). The exact filters used are similar to those used in 7z, as 7z's filters are available in the public domain via the LZMA SDK. xz's RISC-V BCJ filter is its own addition.

The author of lzip claims that the xz format is inadequate for long-term archiving.[8]

All multi-byte values are encoded in little-endian.[9]

Offset
(in bytes) Field Size
(in bytes) Description 0 Header magic number 6 Magic number. Must be FD 37 7A 58 5A 00. 6 Flags 2 Flags. The first byte and the four most significant bits of the second byte must be zero (reserved for future use).

The type of check (last field in the block structure) is encoded in the four least significant bits of the second byte:

Value Size
(in bytes) Description 0 0 None 1 4 CRC-32 2 4 Reserved 3 4 Reserved 4 8 CRC-64 5 8 Reserved 6 8 Reserved 7 16 Reserved 8 16 Reserved 9 16 Reserved 10 32 SHA-256 11 32 Reserved 12 32 Reserved 13 64 Reserved 14 64 Reserved 15 64 Reserved 8 Header CRC32 4 CRC-32 of the flags field. Used to distinguish between corrupted a file and unsupported flags (i.e. reserved bit set). 12 Blocks Varies Sequence of zero or more blocks. Varies Index Varies See index below. Varies Footer CRC32 4 CRC-32 of the flags and backward size. Varies Backward Size 4 Size of the index field. Varies Flags 2 Copy of the flags field above. Varies Footer magic number 2 Magic number. Must be 59 5A. Offset
(in bytes) Field Size
(in bytes) Description 0 Header size 1 Size of the header. Note: real_header_size = (encoded_header_size + 1) * 4. 1 Flags 1 Flags 2 Compressed size 0 or varies Size of the compressed data. Present if bit 6 of the flags is set. Encoded as a variable-length integer. Varies Uncompressed size 0 or varies Size of the block after decompression. Present if bit 7 of the flags is set. Encoded as a variable-length integer. Varies Filter flags Varies Sequence of filter flags. The amount is encoded in bits 0-1 of the flags. Varies Header padding Varies As many null bytes as needed to make the header (i.e. fields before the compressed data) have the size specified in the header size field. Varies CRC32 4 CRC-32 of all bytes in the block up to (not including) this field. Varies Compressed data Varies The compressed data. Varies Block padding 0, 1, 2 or 3 0-3 null bytes to make the size of the block a multiple of 4. Varies Check 0, 4, 8, or 32 Error-detecting mechanism calculated from the data before compression. The type of check is encoded in the flags of the stream structure. Development and adoption[edit]

Development of XZ Utils took place within the Tukaani Project, a small group of developers who once maintained a Linux distribution based on Slackware. The chosen name "XZ" is not an abbreviation but instead appears to be a random given name for the data compressors, as there is no mention anywhere in the official specification on the meaning of "XZ".[10] The .xz file format specification version 1.0.0 was officially released in January 2009.[11]

All of the source code for xz and liblzma has been released into the public domain. The XZ Utils source distribution additionally includes some optional scripts and an example program that are subject to various versions of the GNU General Public License (GPL).[1] The resulting software xz and liblzma binaries are public domain, unless the optional LGPL getopt implementation is incorporated.[12]

Binaries are available for FreeBSD, NetBSD, Linux systems, Microsoft Windows, and FreeDOS. A number of Linux distributions, including Fedora, Slackware, Ubuntu, and Debian use xz for compressing their software packages. Arch Linux previously used xz to compress packages,[13] but as of 27 December 2019, packages are compressed with Zstandard compression.[14] Fedora Linux also switched to compressing its RPM packages with Zstandard with Fedora Linux 31.[15] The GNU FTP archive also uses xz.

On 29 March 2024, Andres Freund, a PostgreSQL developer working at Microsoft, announced that he had found a backdoor in XZ Utils, impacting versions 5.6.0 and 5.6.1. Malicious code for setting up the backdoor had been hidden in compressed test files, and the configure script in the tar files was modified to trigger the hidden code. Freund started his investigation because "After observing a few odd symptoms around liblzma (part of the xz package)" as he found that ssh logins using sshd were "taking a lot of CPU, valgrind errors".[16] The vulnerability received a Common Vulnerability Scoring System (CVSS) score of 10 (the highest).[17]

  1. ^ a b Licensing on tukaani.org "The most interesting parts of XZ Utils (e.g. liblzma) are in the public domain. You can do whatever you want with the public domain parts. Some parts of XZ Utils (e.g. build system and some utilities) are under different free software licenses such as GNU LGPLv2.1, GNU GPLv2, or GNU GPLv3."
  2. ^ LZMA Util, retrieved 25 January 2011
  3. ^ a b "XZ Utils Release Notes". git.tukaani.org.
  4. ^ Vivek, Gite. "How to compress the whole directory using xz and tar in Linux". For instance, I compressed a directory having 37M size using both xz and zip. The zip file size was 31M, while the xz file was 16M after compression
  5. ^ Henry-Stocker, Sandra (12 December 2017). "How to squeeze the most out of Linux file compression". Network World. Retrieved 9 February 2020.
  6. ^ "Gzip vs Bzip2 vs XZ Performance Comparison". RootUsers. 16 September 2015. Retrieved 9 February 2020.
  7. ^ a b "xz, unxz, xzcat, lzma, unlzma, lzcat – Compress or decompress .xz and .lzma files". Linux Manpages Online.
  8. ^ Diaz Diaz, Antonio (3 April 2025). "Xz format inadequate for long-term archiving". nongnu.org. Retrieved 4 April 2025.
  9. ^ "The .xz File Format". 1.2. Multibyte Integers.
  10. ^ "Official XZ Specification". tukaani.org. Lasse Collin. Retrieved 8 October 2024.
  11. ^ Lasse Collin (28 January 2009). "News: The .xz file format specification version 1.0.0 is now officially released".
  12. ^ "In what cases is the output of a GPL program covered by the GPL too?". GNU.org. Retrieved 21 August 2019.
  13. ^ Pierre Schmitz (23 March 2010). "News: Switching to xz compression for new packages".
  14. ^ "Arch Linux - News: Now using Zstandard instead of xz for package compression". www.archlinux.org. Retrieved 7 January 2020.
  15. ^ Mach, Daniel. "Changes/Switch RPMs to zstd compression". Fedora Project Wiki. Retrieved 30 March 2024.
  16. ^ "oss-security - backdoor in upstream xz/liblzma leading to ssh server compromise". www.openwall.com. Retrieved 8 April 2024.
  17. ^ "A backdoor in xz". LWN.net. Retrieved 30 March 2024.

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4