11 KiB

Raw Permalink Blame History

std.compress - Compression API Reference (Zig 0.16.0)

Compression and decompression algorithms. Zig 0.16 adds Deflate compression, simplifies decompression, and continues migrating compression APIs to std.Io.Reader / std.Io.Writer.

Primary Zig 0.16 release-note source: https://ziglang.org/download/0.16.0/release-notes.html

The examples below use Zig 0.16's concrete std.Io.Reader / std.Io.Writer interfaces rather than removed generic-reader or fixed-buffer-stream patterns.

Module Structure
DEFLATE (gzip/zlib)
Zstandard
LZMA
LZMA2
XZ
Common Patterns

Module Structure

std.compress.flate      // DEFLATE: gzip, zlib, raw deflate
std.compress.zstd       // Zstandard compression
std.compress.lzma       // LZMA compression
std.compress.lzma2      // LZMA2 compression
std.compress.xz         // XZ format (LZMA2 container)

DEFLATE (gzip/zlib)

DEFLATE compression with gzip, zlib, or raw containers. Defined in RFC 1951 (deflate), RFC 1950 (zlib), RFC 1952 (gzip).

Container Types

const Container = enum {
    raw,   // No header/footer, raw deflate stream
    gzip,  // gzip header (10+ bytes) + deflate + CRC32 + size footer (8 bytes)
    zlib,  // zlib header (2 bytes) + deflate + Adler32 footer (4 bytes)
};

Decompression

const flate = std.compress.flate;

// Decompress gzip data
var input: std.Io.Reader = .fixed(compressed_data);
var output: std.Io.Writer.Allocating = .init(allocator);
defer output.deinit();

var decompress: flate.Decompress = .init(&input, .gzip, &.{});
_ = try decompress.reader.streamRemaining(&output.writer);

const decompressed = output.written();

With History Buffer

For streaming decompression with backref support:

var buffer: [flate.max_window_len]u8 = undefined;
var decompress: flate.Decompress = .init(&input, .zlib, &buffer);

Decompress Constants

flate.max_window_len  // 65536 - Maximum window size (32768 * 2)
flate.history_len     // 32768 - History buffer length

Compression

const flate = std.compress.flate;

var output: std.Io.Writer.Allocating = .init(allocator);
defer output.deinit();

var buffer: [flate.max_window_len]u8 = undefined;
var compress: flate.Compress = try .init(
    &output.writer,
    &buffer,
    .gzip,
    .default,
);

try compress.writer.writeAll(data);
try compress.finish();

const compressed = output.written();

Compression Levels

const Options = struct {
    // Options are parameter sets rather than an enum. Levels 1 through 9 are
    // available; these public constants select common presets:
    level_1: Options,
    level_4,  // Fastest
    level_5,
    level_6,  // Default
    level_7,
    level_8,
    level_9,  // Best compression

    fastest,  // Alias for level_1
    default,  // Alias for level_6
    best,     // Alias for level_9
};

Huffman-Only Compression

flate.Compress.Huffman exposes public initialization and a writer that skips LZ77 match searching. However, in the installed Zig 0.16 source its end-of-stream finish method is private. External code therefore cannot complete the container lifecycle through the public API alone. Treat it as an implementation-facing type in this release rather than a standalone archive-writing recipe; use flate.Compress when a complete public compression lifecycle is required.

Zstandard

Zstandard (zstd) decompression. High compression ratio with fast decompression.

Decompression

const zstd = std.compress.zstd;

var input: std.Io.Reader = .fixed(compressed_data);
var output: std.Io.Writer.Allocating = .init(allocator);
defer output.deinit();

var decompress: zstd.Decompress = .init(&input, &.{}, .{});
_ = try decompress.reader.streamRemaining(&output.writer);

const decompressed = output.written();

With Custom Window Size

var buffer: [zstd.default_window_len + zstd.block_size_max]u8 = undefined;
var decompress: zstd.Decompress = .init(&input, &buffer, .{
    .window_len = zstd.default_window_len,
    .verify_checksum = false,  // Not yet implemented
});

Zstd Constants

zstd.default_window_len  // 8 * 1024 * 1024 (8 MB)
zstd.block_size_max      // 1 << 17 (128 KB)

Options

pub const Options = struct {
    verify_checksum: bool = false,  // Not yet implemented
    window_len: u32 = zstd.default_window_len,
};

LZMA

LZMA decompression with streaming reader interface.

Decompression

const lzma = std.compress.lzma;

var decoder_buffer = try allocator.alloc(u8, 4096);
errdefer allocator.free(decoder_buffer);
var decompress: lzma.Decompress = try .initOptions(
    &reader,
    allocator,
    decoder_buffer,
    .{},
    128 * 1024 * 1024,
);
defer decompress.deinit();

_ = try decompress.reader.streamRemaining(&output.writer);

With Options

var decompress: lzma.Decompress = try .initOptions(
    &reader,
    allocator,
    decoder_buffer,
    .{ .allow_incomplete = false },
    128 * 1024 * 1024,
);

Decompress Type

// Decompress embeds `reader: std.Io.Reader` and owns the caller-supplied
// buffer after init. `deinit` frees that buffer unless `takeBuffer` first
// reclaims it. `initParams` and `initOptions` are the construction paths.

LZMA2

LZMA2 decompression (improved LZMA with better streaming support).

Decompression

const lzma2 = std.compress.lzma2;

var input: std.Io.Reader = .fixed(compressed_data);
var output: std.Io.Writer.Allocating = .init(allocator);
defer output.deinit();
var decode = try lzma2.Decode.init(allocator);
defer decode.deinit(allocator);
_ = try decode.decompress(&input, &output);

XZ

XZ format decompression (LZMA2 in a container with checksums).

Decompression

const xz = std.compress.xz;

var decoder_buffer = try allocator.alloc(u8, 4096);
errdefer allocator.free(decoder_buffer);
var decompress: xz.Decompress = try .init(&reader, allocator, decoder_buffer);
defer decompress.deinit();

_ = try decompress.reader.streamRemaining(&output.writer);

Check Types

XZ parses these integrity-check identifiers, but Zig 0.16 does not implement full XZ block-check verification. Do not treat successful decoding as verification of CRC32, CRC64, or SHA-256 block checks:

pub const Check = enum(u4) {
    none = 0x00,
    crc32 = 0x01,
    crc64 = 0x04,
    sha256 = 0x0A,
    _,
};

Common Patterns

Decompress gzip File

fn decompressGzip(allocator: Allocator, compressed: []const u8) ![]u8 {
    const flate = std.compress.flate;

    var input: std.Io.Reader = .fixed(compressed);
    var output: std.Io.Writer.Allocating = .init(allocator);
    errdefer output.deinit();

    var decompress: flate.Decompress = .init(&input, .gzip, &.{});
    _ = try decompress.reader.streamRemaining(&output.writer);

    return output.toOwnedSlice();
}

Decompress zlib Data

fn decompressZlib(allocator: Allocator, compressed: []const u8) ![]u8 {
    const flate = std.compress.flate;

    var input: std.Io.Reader = .fixed(compressed);
    var output: std.Io.Writer.Allocating = .init(allocator);
    errdefer output.deinit();

    var decompress: flate.Decompress = .init(&input, .zlib, &.{});
    _ = try decompress.reader.streamRemaining(&output.writer);

    return output.toOwnedSlice();
}

Decompress Zstandard

fn decompressZstd(allocator: Allocator, compressed: []const u8) ![]u8 {
    const zstd = std.compress.zstd;

    var input: std.Io.Reader = .fixed(compressed);
    var output: std.Io.Writer.Allocating = .init(allocator);
    errdefer output.deinit();

    var decompress: zstd.Decompress = .init(&input, &.{}, .{});
    _ = try decompress.reader.streamRemaining(&output.writer);

    return output.toOwnedSlice();
}

Stream Decompression to File

fn decompressToFile(
    io: std.Io,
    input_path: []const u8,
    output_path: []const u8,
    container: std.compress.flate.Container,
) !void {
    const flate = std.compress.flate;

    const input_file = try std.Io.Dir.cwd().openFile(io, input_path, .{});
    defer input_file.close(io);

    const output_file = try std.Io.Dir.cwd().createFile(io, output_path, .{});
    defer output_file.close(io);

    var input_buf: [4096]u8 = undefined;
    var input_reader = input_file.reader(io, &input_buf);

    var output_buf: [4096]u8 = undefined;
    var output_writer = output_file.writer(io, &output_buf);

    var decompress: flate.Decompress = .init(&input_reader.interface, container, &.{});
    _ = try decompress.reader.streamRemaining(&output_writer.interface);
    try output_writer.interface.flush();
}

Detect Compression Format

fn detectFormat(data: []const u8) ?enum { gzip, zlib, zstd, xz } {
    if (data.len < 2) return null;

    // gzip: 0x1f 0x8b
    if (data[0] == 0x1f and data[1] == 0x8b) return .gzip;

    // zlib: CMF byte with CM=8, CINFO<=7
    const cmf = data[0];
    if ((cmf & 0x0f) == 8 and (cmf >> 4) <= 7) {
        // Check FCHECK makes header divisible by 31
        const header: u16 = (@as(u16, data[0]) << 8) | data[1];
        if (header % 31 == 0) return .zlib;
    }

    // zstd: magic 0xFD2FB528
    if (data.len >= 4) {
        const magic = std.mem.readInt(u32, data[0..4], .little);
        if (magic == 0xFD2FB528) return .zstd;
    }

    // xz: magic 0xFD377A585A00
    if (data.len >= 6) {
        if (std.mem.eql(u8, data[0..6], &.{ 0xFD, '7', 'z', 'X', 'Z', 0x00 })) return .xz;
    }

    return null;
}

Error Types

DEFLATE Errors

pub const Error = Container.Error || error{
    InvalidCode,
    InvalidMatch,
    WrongStoredBlockNlen,
    InvalidBlockType,
    InvalidDynamicBlockHeader,
    ReadFailed,
    OversubscribedHuffmanTree,
    IncompleteHuffmanTree,
    MissingEndOfBlockCode,
    EndOfStream,
};

pub const Container.Error = error{
    BadGzipHeader,
    BadZlibHeader,
    WrongGzipChecksum,
    WrongGzipSize,
    WrongZlibChecksum,
};

Zstandard Errors

pub const Error = error{
    BadMagic,
    BlockOversize,
    ChecksumFailure,
    ContentOversize,
    DictionaryIdFlagUnsupported,
    EndOfStream,
    HuffmanTreeIncomplete,
    InvalidBitStream,
    MalformedAccuracyLog,
    MalformedBlock,
    MalformedCompressedBlock,
    MalformedFrame,
    MalformedFseBits,
    MalformedFseTable,
    MalformedHuffmanTree,
    MalformedLiteralsHeader,
    MalformedLiteralsLength,
    MalformedLiteralsSection,
    MalformedSequence,
    MissingStartBit,
    OutputBufferUndersize,
    InputBufferUndersize,
    ReadFailed,
    RepeatModeFirst,
    ReservedBitSet,
    ReservedBlock,
    SequenceBufferUndersize,
    TreelessLiteralsFirst,
    UnexpectedEndOfLiteralStream,
    WindowOversize,
    WindowSizeUnknown,
};

Supported Features

DEFLATE (flate):

Decompression: gzip, zlib, raw deflate
Compression: gzip, zlib, raw deflate (levels 1-9)
Streaming with history buffer

Zstandard (zstd):

Decompression only
Skippable frames
Configurable window size
Dictionary support: Not implemented

LZMA/LZMA2:

Decompression only
Streaming interface
Memory limit configuration

XZ:

Decompression only
CRC32/CRC64/SHA256 check IDs are parsed; full block-check verification is incomplete
Multiple block support

11 KiB Raw Permalink Blame History

std.compress - Compression API Reference (Zig 0.16.0)

Table of Contents

Module Structure

DEFLATE (gzip/zlib)

Container Types

Decompression

With History Buffer

Decompress Constants

Compression

Compression Levels

Huffman-Only Compression

Zstandard

Decompression

With Custom Window Size

Zstd Constants

Options

LZMA

Decompression

With Options

Decompress Type

LZMA2

Decompression

XZ

Decompression

Check Types

Common Patterns

Decompress gzip File

Decompress zlib Data

Decompress Zstandard

Stream Decompression to File

Detect Compression Format

Error Types

DEFLATE Errors

Zstandard Errors

Supported Features

11 KiB

Raw Permalink Blame History