compress.zstd

import compress

compress exposes this as compress.zstd, so import compress is enough and the names are called as compress.zstd.*. import compress.zstd reaches the same definitions directly.

This is the Zstd submodule for the compress module. It implements zstd compression levels -8 through 4 (Fast and DFast strategies), targeting high-speed compression for data transfers. It produces standard zstd frames decompressible by any compliant decoder.

Functions

compress()

compress.zstd.compress(data, level) -> bytes

Compress data using the default options for Zstd.

Parameters

• data (bytes|string)
• level (?number)

Returns bytes

decompress()

compress.zstd.decompress(data) -> bytes

Decompress a Zstd compressed data.

Parameters

• data (bytes|string)

Returns bytes

Classes

ZstdEncoder

class compress.zstd.ZstdEncoder

Streaming zstd compressor implemention. This encoder supports levels -8 through 4 for fast transfer pipelines.

Buffers input until a full block (128 KiB) is ready, then compresses and writes it to the underlying writer. Call finish() to flush the final block, write the content checksum, and recover the writer.

Internal buffers (hash tables, sequence scratch, block encoder workspace) are allocated once and reused across blocks. To reuse them across multiple frames, call reset() instead of finish.

Example:

%> import compress.zstd
%> var encoder = zstd.ZstdEncoder()
%> encoder.write('hello world')
11
%> encoder.finish()
(28 b5 2f fd 04 00 59 00 00 68 65 6c 6c 6f 20 77 6f 72 6c 64 68 69 1e b2)

The normal match finder operates within a sliding window (512 KiB at L1). LDM (Long Distance Matching) finds matches at distances up to 1 << window_log bytes by sampling positions into a separate hash table. Useful for data with long-range repeats: log files, database dumps, source archives.

window_log controls the maximum match distance (24 = 16 MiB, 27 = 128 MiB). The main cost is memory: the encoder allocates an 8 MiB LDM hash table plus a window buffer of 1 << window_log bytes, and the decoder allocates a window buffer of the same size (declared in the frame header). At window_log 27 that is ~136 MiB on each side. On data without long-range repeats, LDM adds overhead with no ratio benefit.

ZstdEncoder own persistent hash tables and workspace buffers. Call reset() to start a new frame while reusing all allocations.

Constructor

compress.zstd.ZstdEncoder(level, window_log, ldm)

Creates a new streaming Zstd encoder at the given level (-8..=4).

Negative levels (-8 through -1) unlocks zstd’s fastest compression tiers. They can be useful when throughput matters more than ratio.

Level 0 maps to the module’s default which is currently level 1.

Positive levels spend more match-finding work for better ratios while staying in the Fast/DFast range.

Parameters

• level (?number) — Default 1
• window_log (?number) — Default 10
• ldm (?bool) — Default false

ZstdEncoder.finish()

compress.zstd.ZstdEncoder.finish() -> bytes

Flushes remaining data, writes the content checksum, and returns the inner byte stream.

Returns bytes

Note: Once finish is called, the encoder can no longer be reused.

ZstdEncoder.close()

compress.zstd.ZstdEncoder.close() -> bytes

Same as finish() for file API compartibility.

Returns bytes

Note: Once finish is called, the encoder can no longer be reused.

ZstdEncoder.reset()

compress.zstd.ZstdEncoder.reset() -> bytes

Finishes the current frame and installs byte stream for the next one.

Returns the previous byte stream containing the completed frame. All internal buffers (hash tables, workspace, block scratch) stay allocated and are reused for the next frame.

Returns bytes

ZstdEncoder.write()

compress.zstd.ZstdEncoder.write(data) -> number

Writes data into this encoder’s byte stream, returning how many bytes were written.

Parameters

• data (bytes|string)

Returns number

ZstdEncoder.write_all()

compress.zstd.ZstdEncoder.write_all(data)

Attempts to write an entire data into this encoder’s byte stream.

Parameters

• data (bytes|string)

ZstdEncoder.flush()

compress.zstd.ZstdEncoder.flush()

Flushes this output stream, ensuring that all intermediately buffered contents reach their destination.

ZstdDecoder

class compress.zstd.ZstdDecoder

Streaming zstd decompressor implemention. This decoder reads standard zstd blocks and frames produced at any zstd compression level.

Wraps a byte stream of compressed data and yields decompressed bytes. Supports multi-frame streams and skippable frames.

Example:

%> import compress.zstd
%> var data = zstd.compress('hello world')
%> data
(28 b5 2f fd 24 0b 59 00 00 68 65 6c 6c 6f 20 77 6f 72 6c 64 68 69 1e b2)
%> 
%> var decoder = zstd.ZstdDecoder(data)
%> decoder.read_as_string()
hello world

ZstdDecoder own persistent hash tables and workspace buffers. Call reset() to start a new frame while reusing all allocations.

This decoder is level-independent and supports standard zstd frames produced by all zstd compression levels.

Constructor

compress.zstd.ZstdDecoder(source)

Creates a Zstd decoder

Parameters

• source (bytes)

ZstdDecoder.reset()

compress.zstd.ZstdDecoder.reset(new_source)

Installs a new data source for the next frame, keeping all internal buffers allocated.

Parameters

• new_source (bytes)

ZstdDecoder.close()

compress.zstd.ZstdDecoder.close()

Closes the decoder by resetting it into an empty stream.

ZstdDecoder.read()

compress.zstd.ZstdDecoder.read(length) -> bytes

Reads some bytes up to the amount of bytes specified by length from the current source. Returns an empty byte stream when there is no more data to read.

This function does not block or wait waiting for data, but reads as much data as is available to read when it runs.

Parameters

• length (number)

Returns bytes

ZstdDecoder.read_exact()

compress.zstd.ZstdDecoder.read_exact(length) -> bytes

Reads the exact number of bytes from the buffer. This method will raise an error if it encounters an unexpected EOF (end of file) or there are insufficient data to read to complete the required number of bytes.

Parameters

• length (number)

Returns bytes

ZstdDecoder.read_all()

compress.zstd.ZstdDecoder.read_all() -> bytes

Reads all remaining bytes until EOF is encountered in the source.

Returns bytes

ZstdDecoder.read_as_string()

compress.zstd.ZstdDecoder.read_as_string() -> string

Reads all remaining bytes until EOF is encountered in the source and returns the data read as a string instead of a byte stream.

Returns string

2021, Richard Ore and Zuri contributors