10 KiB

Raw Blame History

std.compress - Compression API Reference (Zig 0.16.0)

Compression and decompression algorithms. Zig 0.16 adds Deflate compression, simplifies decompression, and continues migrating compression APIs to std.Io.Reader / std.Io.Writer.

Primary Zig 0.16 release-note source: https://ziglang.org/download/0.16.0/release-notes.html

Old examples below may use removed std.io.GenericReader or fixedBufferStream patterns. Translate them to std.Io.Reader / std.Io.Writer before using them in Zig 0.16 code.

Module Structure
DEFLATE (gzip/zlib)
Zstandard
LZMA
LZMA2
XZ
Common Patterns

Module Structure

std.compress.flate      // DEFLATE: gzip, zlib, raw deflate
std.compress.zstd       // Zstandard compression
std.compress.lzma       // LZMA compression
std.compress.lzma2      // LZMA2 compression
std.compress.xz         // XZ format (LZMA2 container)

DEFLATE (gzip/zlib)

DEFLATE compression with gzip, zlib, or raw containers. Defined in RFC 1951 (deflate), RFC 1950 (zlib), RFC 1952 (gzip).

Container Types

const Container = enum {
    raw,   // No header/footer, raw deflate stream
    gzip,  // gzip header (10+ bytes) + deflate + CRC32 + size footer (8 bytes)
    zlib,  // zlib header (2 bytes) + deflate + Adler32 footer (4 bytes)
};

Decompression

const flate = std.compress.flate;

// Decompress gzip data
var input: std.Io.Reader = .fixed(compressed_data);
var output: std.Io.Writer.Allocating = .init(allocator);
defer output.deinit();

var decompress: flate.Decompress = .init(&input, .gzip, &.{});
_ = try decompress.reader.streamRemaining(&output.writer);

const decompressed = output.written();

With History Buffer

For streaming decompression with backref support:

var buffer: [flate.max_window_len]u8 = undefined;
var decompress: flate.Decompress = .init(&input, .zlib, &buffer);

Decompress Constants

flate.max_window_len  // 65536 - Maximum window size (32768 * 2)
flate.history_len     // 32768 - History buffer length

Compression

const flate = std.compress.flate;

var output: std.Io.Writer.Allocating = .init(allocator);
defer output.deinit();

var buffer: [flate.max_window_len]u8 = undefined;
var compress: flate.Compress = .init(&output.writer, &buffer, .{
    .level = .default,
    .container = .gzip,
});

try compress.writer.writeAll(data);
try compress.end();

const compressed = output.written();

Compression Levels

const Level = enum {
    level_4,  // Fastest
    level_5,
    level_6,  // Default
    level_7,
    level_8,
    level_9,  // Best compression

    fast,     // Alias for level_4
    default,  // Alias for level_6
    best,     // Alias for level_9
};

Huffman-Only Compression

For faster compression without LZ77 match searching:

const HuffmanEncoder = flate.HuffmanEncoder;
// Used internally for Huffman-only encoding (bigger output, faster compression)

Zstandard

Zstandard (zstd) decompression. High compression ratio with fast decompression.

Decompression

const zstd = std.compress.zstd;

var input: std.Io.Reader = .fixed(compressed_data);
var output: std.Io.Writer.Allocating = .init(allocator);
defer output.deinit();

var decompress: zstd.Decompress = .init(&input, &.{}, .{});
_ = try decompress.reader.streamRemaining(&output.writer);

const decompressed = output.written();

With Custom Window Size

var buffer: [zstd.default_window_len + zstd.block_size_max]u8 = undefined;
var decompress: zstd.Decompress = .init(&input, &buffer, .{
    .window_len = zstd.default_window_len,
    .verify_checksum = false,  // Not yet implemented
});

Zstd Constants

zstd.default_window_len  // 8 * 1024 * 1024 (8 MB)
zstd.block_size_max      // 1 << 17 (128 KB)

Options

pub const Options = struct {
    verify_checksum: bool = false,  // Not yet implemented
    window_len: u32 = zstd.default_window_len,
};

LZMA

LZMA decompression with streaming reader interface.

Decompression

const lzma = std.compress.lzma;

var decompress = try lzma.decompress(allocator, reader);
defer decompress.deinit();

var buf: [4096]u8 = undefined;
while (true) {
    const n = try decompress.read(&buf);
    if (n == 0) break;
    // Process buf[0..n]
}

With Options

var decompress = try lzma.decompressWithOptions(allocator, reader, .{
    .memlimit = 128 * 1024 * 1024,  // 128 MB memory limit
});

Decompress Type

pub fn Decompress(comptime ReaderType: type) type {
    return struct {
        pub const Reader = std.io.GenericReader(*Self, Error, read);

        pub fn init(allocator: Allocator, source: ReaderType, params: Params, memlimit: ?usize) !Self;
        pub fn deinit(self: *Self) void;
        pub fn reader(self: *Self) Reader;
        pub fn read(self: *Self, output: []u8) Error!usize;
    };
}

LZMA2

LZMA2 decompression (improved LZMA with better streaming support).

Decompression

const lzma2 = std.compress.lzma2;

var output = std.ArrayList(u8).empty;
defer output.deinit(allocator);

var stream = std.io.fixedBufferStream(compressed_data);
try lzma2.decompress(allocator, stream.reader(), output.writer(allocator));

XZ

XZ format decompression (LZMA2 in a container with checksums).

Decompression

const xz = std.compress.xz;

var decompress = try xz.decompress(allocator, reader);
defer decompress.deinit();

var buf: [4096]u8 = undefined;
while (true) {
    const n = try decompress.read(&buf);
    if (n == 0) break;
    // Process buf[0..n]
}

Check Types

XZ supports multiple integrity check types:

pub const Check = enum(u4) {
    none = 0x00,
    crc32 = 0x01,
    crc64 = 0x04,
    sha256 = 0x0A,
    _,
};

Common Patterns

Decompress gzip File

fn decompressGzip(allocator: Allocator, compressed: []const u8) ![]u8 {
    const flate = std.compress.flate;

    var input: std.Io.Reader = .fixed(compressed);
    var output: std.Io.Writer.Allocating = .init(allocator);
    errdefer output.deinit();

    var decompress: flate.Decompress = .init(&input, .gzip, &.{});
    _ = try decompress.reader.streamRemaining(&output.writer);

    return output.toOwnedSlice();
}

Decompress zlib Data

fn decompressZlib(allocator: Allocator, compressed: []const u8) ![]u8 {
    const flate = std.compress.flate;

    var input: std.Io.Reader = .fixed(compressed);
    var output: std.Io.Writer.Allocating = .init(allocator);
    errdefer output.deinit();

    var decompress: flate.Decompress = .init(&input, .zlib, &.{});
    _ = try decompress.reader.streamRemaining(&output.writer);

    return output.toOwnedSlice();
}

Decompress Zstandard

fn decompressZstd(allocator: Allocator, compressed: []const u8) ![]u8 {
    const zstd = std.compress.zstd;

    var input: std.Io.Reader = .fixed(compressed);
    var output: std.Io.Writer.Allocating = .init(allocator);
    errdefer output.deinit();

    var decompress: zstd.Decompress = .init(&input, &.{}, .{});
    _ = try decompress.reader.streamRemaining(&output.writer);

    return output.toOwnedSlice();
}

Stream Decompression to File

fn decompressToFile(
    input_path: []const u8,
    output_path: []const u8,
    container: std.compress.flate.Container,
) !void {
    const flate = std.compress.flate;

    const input_file = try std.fs.cwd().openFile(input_path, .{});
    defer input_file.close();

    const output_file = try std.fs.cwd().createFile(output_path, .{});
    defer output_file.close();

    var input_buf: [4096]u8 = undefined;
    var input_reader = input_file.reader(&input_buf);

    var output_buf: [4096]u8 = undefined;
    var output_writer = output_file.writer(&output_buf);

    var decompress: flate.Decompress = .init(&input_reader.interface, container, &.{});
    _ = try decompress.reader.streamRemaining(&output_writer.interface);
    try output_writer.interface.flush();
}

Detect Compression Format

fn detectFormat(data: []const u8) ?enum { gzip, zlib, zstd, xz } {
    if (data.len < 2) return null;

    // gzip: 0x1f 0x8b
    if (data[0] == 0x1f and data[1] == 0x8b) return .gzip;

    // zlib: CMF byte with CM=8, CINFO<=7
    const cmf = data[0];
    if ((cmf & 0x0f) == 8 and (cmf >> 4) <= 7) {
        // Check FCHECK makes header divisible by 31
        const header: u16 = (@as(u16, data[0]) << 8) | data[1];
        if (header % 31 == 0) return .zlib;
    }

    // zstd: magic 0xFD2FB528
    if (data.len >= 4) {
        const magic = std.mem.readInt(u32, data[0..4], .little);
        if (magic == 0xFD2FB528) return .zstd;
    }

    // xz: magic 0xFD377A585A00
    if (data.len >= 6) {
        if (std.mem.eql(u8, data[0..6], &.{ 0xFD, '7', 'z', 'X', 'Z', 0x00 })) return .xz;
    }

    return null;
}

Error Types

DEFLATE Errors

pub const Error = Container.Error || error{
    InvalidCode,
    InvalidMatch,
    WrongStoredBlockNlen,
    InvalidBlockType,
    InvalidDynamicBlockHeader,
    ReadFailed,
    OversubscribedHuffmanTree,
    IncompleteHuffmanTree,
    MissingEndOfBlockCode,
    EndOfStream,
};

pub const Container.Error = error{
    BadGzipHeader,
    BadZlibHeader,
    WrongGzipChecksum,
    WrongGzipSize,
    WrongZlibChecksum,
};

Zstandard Errors

pub const Error = error{
    BadMagic,
    BlockOversize,
    ChecksumFailure,
    ContentOversize,
    DictionaryIdFlagUnsupported,
    EndOfStream,
    HuffmanTreeIncomplete,
    InvalidBitStream,
    MalformedAccuracyLog,
    MalformedBlock,
    MalformedCompressedBlock,
    MalformedFrame,
    MalformedFseBits,
    MalformedFseTable,
    MalformedHuffmanTree,
    MalformedLiteralsHeader,
    MalformedLiteralsLength,
    MalformedLiteralsSection,
    MalformedSequence,
    MissingStartBit,
    OutputBufferUndersize,
    InputBufferUndersize,
    ReadFailed,
    RepeatModeFirst,
    ReservedBitSet,
    ReservedBlock,
    SequenceBufferUndersize,
    TreelessLiteralsFirst,
    UnexpectedEndOfLiteralStream,
    WindowOversize,
    WindowSizeUnknown,
};

Supported Features

DEFLATE (flate):

Decompression: gzip, zlib, raw deflate
Compression: gzip, zlib, raw deflate (levels 4-9)
Streaming with history buffer

Zstandard (zstd):

Decompression only
Skippable frames
Configurable window size
Dictionary support: Not implemented

LZMA/LZMA2:

Decompression only
Streaming interface
Memory limit configuration

XZ:

Decompression only
CRC32/CRC64/SHA256 integrity checks
Multiple block support

10 KiB Raw Blame History

std.compress - Compression API Reference (Zig 0.16.0)

Table of Contents

Module Structure

DEFLATE (gzip/zlib)

Container Types

Decompression

With History Buffer

Decompress Constants

Compression

Compression Levels

Huffman-Only Compression

Zstandard

Decompression

With Custom Window Size

Zstd Constants

Options

LZMA

Decompression

With Options

Decompress Type

LZMA2

Decompression

XZ

Decompression

Check Types

Common Patterns

Decompress gzip File

Decompress zlib Data

Decompress Zstandard

Stream Decompression to File

Detect Compression Format

Error Types

DEFLATE Errors

Zstandard Errors

Supported Features

10 KiB

Raw Blame History