sdlparser-scrap/docs/DEVELOPMENT.md

Development Guide

Guide for contributing to and extending the SDL3 header parser.

Quick Start for Developers

# Clone and build
cd lib/sdl3/parser
zig build

# Run tests
zig build test

# Make changes
# ... edit src/*.zig ...

# Test your changes
zig build test
zig build run -- ../SDL/include/SDL3/SDL_gpu.h --output=test.zig

Project Structure

parser/
├── src/
│   ├── parser.zig              # Main entry point, CLI
│   ├── patterns.zig            # Pattern matching & scanning
│   ├── types.zig               # C to Zig type conversion
│   ├── naming.zig              # Naming conventions
│   ├── codegen.zig             # Zig code generation
│   ├── mock_codegen.zig        # C mock generation
│   └── dependency_resolver.zig # Dependency analysis
├── test/
│   └── (test files)
├── docs/
│   └── (documentation)
└── build.zig

Zig 0.15 API Changes - CRITICAL

This project uses Zig 0.15. Key API changes from 0.14:

ArrayList Changes

Old (0.14):

var list = std.ArrayList(T).init(allocator);
defer list.deinit();
try list.append(item);

New (0.15) - REQUIRED:

var list = std.ArrayList(T){};
defer list.deinit(allocator);
try list.append(allocator, item);

Key Points:

• Initialize with {} or initCapacity()
• All methods take allocator: append(allocator, item)
• Deinit takes allocator: deinit(allocator)

AST Rendering

Old: ast.render(allocator)
New: ast.renderAlloc(allocator)

Adding New Pattern Support

Example: Adding Union Support

1. Add to Declaration union (patterns.zig):

pub const Declaration = union(enum) {
    // ... existing variants
    union_decl: UnionDecl,  // NEW
};

pub const UnionDecl = struct {
    name: []const u8,
    fields: []FieldDecl,
    doc_comment: ?[]const u8,
};

2. Add scanner function (patterns.zig):

fn scanUnion(self: *Scanner) !?UnionDecl {
    // Pattern matching logic
    // Return UnionDecl or null
}

3. Add to scan chain (patterns.zig):

if (try self.scanOpaque()) |opaque_decl| {
    // ...
} else if (try self.scanUnion()) |union_decl| {
    try decls.append(self.allocator, .{ .union_decl = union_decl });
} else if (try self.scanEnum()) |enum_decl| {
    // ...
}

4. Update cleanup code (parser.zig):

defer {
    for (decls) |decl| {
        switch (decl) {
            // ... existing cases
            .union_decl => |u| {
                allocator.free(u.name);
                if (u.doc_comment) |doc| allocator.free(doc);
                for (u.fields) |field| {
                    allocator.free(field.name);
                    allocator.free(field.type_name);
                    if (field.comment) |c| allocator.free(c);
                }
                allocator.free(u.fields);
            },
        }
    }
}

5. Update dependency resolver (dependency_resolver.zig):

// In collectDefinedTypes:
.union_decl => |u| u.name,

// In cloneDeclaration:
.union_decl => |u| .{
    .union_decl = .{
        .name = try allocator.dupe(u8, u.name),
        .fields = try cloneFields(allocator, u.fields),
        .doc_comment = if (u.doc_comment) |doc| 
            try allocator.dupe(u8, doc) else null,
    },
},

// In freeDeclaration:
.union_decl => |u| {
    allocator.free(u.name);
    if (u.doc_comment) |doc| allocator.free(doc);
    for (u.fields) |field| {
        allocator.free(field.name);
        allocator.free(field.type_name);
        if (field.comment) |c| allocator.free(c);
    }
    allocator.free(u.fields);
},

6. Add code generator (codegen.zig):

fn writeUnion(self: *CodeGen, union_decl: patterns.UnionDecl) !void {
    const zig_name = naming.typeNameToZig(union_decl.name);
    
    if (union_decl.doc_comment) |doc| {
        try self.writeDocComment(doc);
    }
    
    try self.output.writer(self.allocator).print(
        "pub const {s} = extern union {{\n", 
        .{zig_name}
    );
    
    for (union_decl.fields) |field| {
        const zig_type = try types.convertType(field.type_name, self.allocator);
        defer self.allocator.free(zig_type);
        
        try self.output.writer(self.allocator).print(
            "    {s}: {s},\n",
            .{field.name, zig_type}
        );
    }
    
    try self.output.appendSlice(self.allocator, "};\n\n");
}

7. Update writeDeclarations (codegen.zig):

switch (decl) {
    // ... existing cases
    .union_decl => |union_decl| try self.writeUnion(union_decl),
}

8. Add tests:

test "parse union" {
    const source =
        \\typedef union SDL_Color {
        \\    Uint32 rgba;
        \\    struct { Uint8 r, g, b, a; };
        \\} SDL_Color;
    ;
    
    var scanner = patterns.Scanner.init(allocator, source);
    const decls = try scanner.scan();
    // ... test expectations
}

Testing Guidelines

Unit Tests

Place tests in test/ or at bottom of source files:

test "descriptive test name" {
    const allocator = std.testing.allocator;
    
    // Setup
    const source = "...";
    var scanner = patterns.Scanner.init(allocator, source);
    
    // Execute
    const result = try scanner.scan();
    defer allocator.free(result);
    
    // Assert
    try std.testing.expectEqual(expected, actual);
}

Integration Tests

Test with real SDL headers:

zig build run -- ../SDL/include/SDL3/SDL_gpu.h --output=test.zig
zig ast-check test.zig

Memory Testing

Always run tests with GPA to detect leaks:

test "my test" {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    defer _ = gpa.deinit();
    const allocator = gpa.allocator();
    
    // Test code using allocator
}

Code Style

Naming

• Functions: camelCase (parseStructField)
• Types: PascalCase (FieldDecl)
• Constants: PascalCase (Declaration)
• Variables: camelCase (decl_name)

Comments

Only comment code that needs clarification:

// Good: Explains WHY
// Must check flags before typedefs - pattern order matters
if (try self.scanFlagTypedef()) |flag_decl| { ... }

// Bad: Explains WHAT (obvious from code)
// Append to list
try list.append(allocator, item);

Error Handling

Use graceful degradation:

// Good: Continue on error
const decl = extractType(source, name) catch |err| {
    std.debug.print("Warning: {}\n", .{err});
    continue;
};

// Bad: Fail immediately (unless truly fatal)
const decl = try extractType(source, name);

Memory Management Rules

Ownership

1. Scanner owns strings during parsing
2. Caller owns result of scan()
3. HashMap owns keys when they're duped
4. Cloned declarations own strings after cloning

Cleanup Pattern

Always use defer for cleanup:

const decls = try scanner.scan();
defer {
    for (decls) |decl| {
        freeDeclDeep(allocator, decl);
    }
    allocator.free(decls);
}

HashMap Keys

Must be owned (not slices into temporary data):

// Wrong:
try map.put(type_str, {});  // type_str might be freed!

// Right:
if (!map.contains(type_str)) {
    const owned = try allocator.dupe(u8, type_str);
    try map.put(owned, {});
}

Common Patterns

Pattern Matching

fn scanSomething(self: *Scanner) !?SomeDecl {
    const start = self.pos;
    const line = try self.readLine();
    defer self.allocator.free(line);
    
    // Check pattern
    if (!std.mem.startsWith(u8, line, "expected_start")) {
        self.pos = start;  // Reset position
        return null;
    }
    
    // Parse and return
    return SomeDecl{ ... };
}

String Building

var buf = std.ArrayList(u8){};
defer buf.deinit(allocator);

try buf.appendSlice(allocator, "pub const ");
try buf.appendSlice(allocator, name);
try buf.appendSlice(allocator, " = ");

return try buf.toOwnedSlice(allocator);

HashMap Usage

var map = std.StringHashMap(void).init(allocator);
defer {
    var it = map.keyIterator();
    while (it.next()) |key| {
        allocator.free(key.*);  // Free owned keys
    }
    map.deinit();
}

// Add items
const owned_key = try allocator.dupe(u8, key);
try map.put(owned_key, {});

Debugging Tips

Print Debugging

std.debug.print("Debug: value = {s}\n", .{value});
std.debug.print("Type: {}\n", .{@TypeOf(variable)});

Memory Leak Detection

Run with GPA and check output:

zig build run -- header.h 2>&1 | grep "memory address"

AST Debugging

Check what Zig thinks is wrong:

zig ast-check generated.zig

Performance Optimization

Guidelines

1. Avoid allocations in hot paths - Use stack when possible
2. Reuse buffers - Clear and reuse instead of allocating new
3. Early exit - Return as soon as answer is known
4. HashMap for lookups - O(1) instead of O(n) searches

Profiling

# Build with profiling
zig build -Drelease-safe

# Run with timing
time zig build run -- large_header.h --output=out.zig

Contributing Workflow

1. Create branch from dev/sdl3-parser
2. Make changes in focused commits
3. Run tests - All must pass
4. Update docs if behavior changes
5. Commit with descriptive message
6. Push and create PR

Commit Message Format

feat: Add union type support

Implements parsing and code generation for C union types.

- Added UnionDecl to Declaration union
- Implemented scanUnion() pattern matcher  
- Added writeUnion() code generator
- Created comprehensive test suite (5 tests)

Results:
- Successfully parses SDL union types
- Generates proper extern unions
- All tests passing

Closes: #123

Test-Driven Development

Recommended workflow:

1. Write test first:

test "parse union type" {
    const source = "typedef union { int x; float y; } SDL_Union;";
    var scanner = patterns.Scanner.init(allocator, source);
    const decls = try scanner.scan();
    
    try testing.expectEqual(@as(usize, 1), decls.len);
    try testing.expect(decls[0] == .union_decl);
}

2. Run test (it will fail)
3. Implement feature until test passes
4. Add more tests for edge cases
5. Refactor if needed

Architecture Decisions

Why Single-File Output?

Alternative: Generate separate file per type

Decision: Single file with dependencies first

Reasons:

• Simpler for users (one import)
• Zig's structural typing handles it
• Type ordering guaranteed
• Less build system complexity

Why On-Demand Resolution?

Alternative: Always parse all includes

Decision: Only parse when missing types detected

Reasons:

• Better performance
• Minimal overhead for self-contained headers
• Users see only relevant dependencies

Why Conservative Error Handling?

Alternative: Fail on any error

Decision: Warn and continue

Reasons:

• Partial success is better than no success
• Users can manually fix issues
• Allows incremental improvement

Extending the Parser

Adding Type Conversions

Edit src/types.zig:

pub fn convertType(c_type: []const u8, allocator: Allocator) ![]const u8 {
    // Check for your pattern first
    if (std.mem.eql(u8, c_type, "MyCustomType")) {
        return try allocator.dupe(u8, "MyZigType");
    }
    
    // Fall through to existing logic
    // ...
}

Adding Naming Rules

Edit src/naming.zig:

pub fn typeNameToZig(c_name: []const u8) []const u8 {
    // Handle special cases
    if (std.mem.eql(u8, c_name, "SDL_bool")) {
        return "Bool";  // Custom mapping
    }
    
    // Default logic
    return stripSDLPrefix(c_name);
}

Adding Pattern Matchers

1. Implement scan*() function in patterns.zig
2. Add to scan chain with proper ordering
3. Update all switch statements
4. Add code generator
5. Write tests

See "Adding New Pattern Support" section above for full example.

Common Issues When Developing

Issue: ArrayList API Changed

Error: error: no field named 'init' in struct 'ArrayList'

Solution: Use Zig 0.15 API (see above)

Issue: HashMap Key Lifetime

Error: Memory corruption or use-after-free

Solution: Always dupe keys before inserting:

const owned = try allocator.dupe(u8, key);
try map.put(owned, {});

Issue: Pattern Matching Order

Error: Wrong scanner function matches

Solution: Order matters! More specific patterns first:

// Correct order:
if (try self.scanFlagTypedef()) { ... }    // Specific
else if (try self.scanTypedef()) { ... }    // General

// Wrong order:
if (try self.scanTypedef()) { ... }         // Too general - catches flags!
else if (try self.scanFlagTypedef()) { ... } // Never reached

Performance Considerations

Current Performance

• Small headers: ~100ms
• Large headers (SDL_gpu.h): ~520ms
• Memory: ~2-5MB peak

Bottlenecks

1. Dependency extraction: 300ms (58% of time)

   • Parsing multiple headers
   • Could cache parsed headers

2. String allocations: Many small allocations

   • Could use arena allocator
   • String interning would help

Optimization Ideas

// Cache parsed headers
var header_cache = std.StringHashMap([]Declaration).init(allocator);

// Use arena for temporary allocations
var arena = std.heap.ArenaAllocator.init(allocator);
defer arena.deinit();
const temp_alloc = arena.allocator();

Resources

Zig Documentation

• Zig Language Reference
• Zig Standard Library

SDL Documentation

• SDL3 API
• SDL3 Headers

Project Documentation

• Architecture - How the parser works
• Dependency Flow - Detailed flow
• Visual Flow - Diagrams

Getting Help

• Check existing tests for examples
• Read Architecture for design
• See Dependency Flow for details
• Review git history for patterns

---

Ready to contribute? Start with the tests, understand the existing patterns, then extend!