# SDL3 Header Parser & Zig Binding Generator ## Overview SDL3's C headers are highly regular and well-structured, making them ideal candidates for automated parsing and Zig binding generation. This document outlines the architecture and implementation plan for a parser that will extract type and function information from SDL3 headers and generate idiomatic Zig bindings. ## Current State The `lib/sdl3/src/` directory contains hand-maintained Zig bindings for SDL3. These bindings demonstrate the target output format that our generator should produce. Key files include: - `gpu.zig` - Comprehensive GPU API bindings (good reference implementation) - `video.zig`, `events.zig`, `init.zig` - Other module bindings - `c.zig` - Direct C imports ## Goals 1. **Parse all 85 SDL3 headers** in `SDL/include/SDL3/` 2. **Extract complete type information**: enums, flags, structs, opaque types, functions 3. **Generate idiomatic Zig bindings** matching the style of existing hand-written bindings 4. **Preserve documentation** from C headers in generated Zig files 5. **Support incremental updates** when SDL3 headers change ## SDL3 Header Patterns ### 1. Opaque Types **C Pattern:** ```c /** * An opaque handle representing a GPU device. * * \since This struct is available since SDL 3.2.0. * * \sa SDL_CreateGPUDevice * \sa SDL_DestroyGPUDevice */ typedef struct SDL_GPUDevice SDL_GPUDevice; ``` **Zig Output:** ```zig pub const GPUDevice = opaque { // Methods will be added here }; ``` ### 2. Enumerations **C Pattern:** ```c /** * Specifies the primitive topology of a graphics pipeline. * * \since This enum is available since SDL 3.2.0. * * \sa SDL_CreateGPUGraphicsPipeline */ typedef enum SDL_GPUPrimitiveType { SDL_GPU_PRIMITIVETYPE_TRIANGLELIST, /**< A series of separate triangles. */ SDL_GPU_PRIMITIVETYPE_TRIANGLESTRIP, /**< A series of connected triangles. */ SDL_GPU_PRIMITIVETYPE_LINELIST, /**< A series of separate lines. */ SDL_GPU_PRIMITIVETYPE_LINESTRIP, /**< A series of connected lines. */ SDL_GPU_PRIMITIVETYPE_POINTLIST /**< A series of separate points. */ } SDL_GPUPrimitiveType; ``` **Zig Output:** ```zig pub const GPUPrimitiveType = enum(c_int) { primitivetypeTrianglelist, //*< A series of separate triangles. */ primitivetypeTrianglestrip, //*< A series of connected triangles. */ primitivetypeLinelist, //*< A series of separate lines. */ primitivetypeLinestrip, //*< A series of connected lines. */ primitivetypePointlist, //*< A series of separate points. */ }; ``` **Naming Convention:** - Remove `SDL_GPU_` prefix - Convert to camelCase starting with lowercase - Example: `SDL_GPU_PRIMITIVETYPE_TRIANGLELIST` → `primitivetypeTrianglelist` ### 3. Flag Types (Bitmasks) **C Pattern:** ```c typedef Uint32 SDL_GPUTextureUsageFlags; #define SDL_GPU_TEXTUREUSAGE_SAMPLER (1u << 0) /**< Texture supports sampling. */ #define SDL_GPU_TEXTUREUSAGE_COLOR_TARGET (1u << 1) /**< Texture is a color render target. */ #define SDL_GPU_TEXTUREUSAGE_DEPTH_STENCIL_TARGET (1u << 2) /**< Texture is a depth stencil target. */ #define SDL_GPU_TEXTUREUSAGE_GRAPHICS_STORAGE_READ (1u << 3) /**< Texture supports storage reads in graphics stages. */ #define SDL_GPU_TEXTUREUSAGE_COMPUTE_STORAGE_READ (1u << 4) /**< Texture supports storage reads in the compute stage. */ #define SDL_GPU_TEXTUREUSAGE_COMPUTE_STORAGE_WRITE (1u << 5) /**< Texture supports storage writes in the compute stage. */ #define SDL_GPU_TEXTUREUSAGE_COMPUTE_STORAGE_SIMULTANEOUS_READ_WRITE (1u << 6) /**< Texture supports reads and writes in the same compute shader. */ ``` **Zig Output:** ```zig pub const GPUTextureUsageFlags = packed struct(u32) { textureusageSampler: bool = false, textureusageColorTarget: bool = false, textureusageDepthStencilTarget: bool = false, textureusageGraphicsStorageRead: bool = false, textureusageComputeStorageRead: bool = false, textureusageComputeStorageWrite: bool = false, textureusageComputeStorageSimultaneousReadWrite: bool = false, pad0: u24 = 0, rsvd: bool = false, }; ``` **Naming Convention:** - Remove common prefix (e.g., `SDL_GPU_TEXTUREUSAGE_`) - Convert to camelCase starting with lowercase - Add padding fields to reach the backing integer size (u32, u64, etc.) - Add `rsvd` field as the high bit for future expansion ### 4. Structures **C Pattern:** ```c /** * A structure specifying the parameters of a graphics pipeline viewport. * * \since This struct is available since SDL 3.2.0. * * \sa SDL_SetGPUViewport */ typedef struct SDL_GPUViewport { float x; /**< The left offset of the viewport. */ float y; /**< The top offset of the viewport. */ float w; /**< The width of the viewport. */ float h; /**< The height of the viewport. */ float min_depth; /**< The minimum depth of the viewport. */ float max_depth; /**< The maximum depth of the viewport. */ } SDL_GPUViewport; ``` **Zig Output:** ```zig pub const GPUViewport = extern struct { x: f32, // The left offset of the viewport. y: f32, // The top offset of the viewport. w: f32, // The width of the viewport. h: f32, // The height of the viewport. min_depth: f32, // The minimum depth of the viewport. max_depth: f32, // The maximum depth of the viewport. }; ``` **Naming Convention:** - Keep field names as-is (already snake_case) - Convert C types to Zig equivalents: - `float` → `f32` - `double` → `f64` - `Uint8` → `u8` - `Uint16` → `u16` - `Uint32` → `u32` - `Uint64` → `u64` - `Sint8` → `i8` - `Sint16` → `i16` - `Sint32` → `i32` - `Sint64` → `i64` - `bool` / `SDL_bool` → `bool` - `size_t` → `usize` - `int` → `c_int` - `char` → `u8` (for single chars) or `[*c]const u8` (for strings) - `void*` → `?*anyopaque` (if nullable) or `*anyopaque` (if non-null) - `const char*` → `[*c]const u8` - `T*` (opaque pointer) → `*T` - `const T*` (opaque pointer) → `*const T` - `T**` (out parameter) → `[*c]*T` ### 5. Constants & Large Enums Some enums in SDL3 have many values and are better represented as individual constants in Zig. **C Pattern:** ```c typedef enum SDL_EventType { SDL_EVENT_FIRST = 0, /**< Unused (do not remove) */ /* Application events */ SDL_EVENT_QUIT = 0x100, /**< User-requested quit */ SDL_EVENT_TERMINATING = 0x101, /**< OS is terminating the app */ // ... many more values } SDL_EventType; ``` **Zig Output (Individual Constants):** ```zig pub const first: u32 = 0; pub const quit: u32 = 256; pub const terminating: u32 = 257; // ... many more constants ``` **Design Decision:** - Large enums (>20 values) that serve as constant collections → individual constants - Small enums that represent a closed set of values → Zig enum - Configuration: Mark certain enums for constant expansion in config ### 6. Functions **C Pattern:** ```c /** * Create a GPU context. * * \param format_flags a bitflag indicating which shader formats the app can * provide. * \param debug_mode enable debug mode properties and validations. * \param name the preferred GPU driver, or NULL to let SDL pick the optimal * driver. * \returns a GPU context on success, or NULL on failure; call SDL_GetError() * for more information. * * \since This function is available since SDL 3.2.0. * * \sa SDL_GetGPUDriver * \sa SDL_DestroyGPUDevice * \sa SDL_GPUSupportsShaderFormats */ extern SDL_DECLSPEC SDL_GPUDevice * SDLCALL SDL_CreateGPUDevice( SDL_GPUShaderFormat format_flags, bool debug_mode, const char *name); ``` **Zig Output (Free Function):** ```zig // SDL_CreateGPUDevice pub inline fn createGPUDevice(format_flags: GPUShaderFormat, debug_mode: bool, name: [*c]const u8) *GPUDevice { return @ptrCast(c.SDL_CreateGPUDevice(@bitCast(format_flags), debug_mode, name)); } ``` **Zig Output (Method on Opaque Type):** ```zig pub const GPUDevice = opaque { // SDL_DestroyGPUDevice pub inline fn destroyGPUDevice(device: *GPUDevice) void { c.SDL_DestroyGPUDevice(@ptrCast(device)); } }; ``` **Function Classification Rules:** 1. Functions taking an opaque type pointer as the first parameter → method on that type 2. Functions that create an opaque type → free function (constructor) 3. All other functions → free functions **Naming Convention:** - Remove `SDL_` prefix - Convert to camelCase - Example: `SDL_CreateGPUDevice` → `createGPUDevice` - For methods, keep the full name but it will be called as `device.destroyGPUDevice(device)` **Cast Handling in Generated Code:** The wrapper functions need to insert appropriate casts: 1. **Opaque pointers:** Use `@ptrCast` ```zig c.SDL_DestroyGPUDevice(@ptrCast(device)) ``` 2. **Enums:** Use `@intFromEnum` (Zig → C) or `@enumFromInt` (C → Zig) ```zig // Zig to C c.SDL_Function(@intFromEnum(my_enum)) // C to Zig return @enumFromInt(c.SDL_Function()) ``` 3. **Flags (packed structs):** Use `@bitCast` ```zig c.SDL_CreateDevice(@bitCast(format_flags)) ``` 4. **Primitive types:** Usually no cast needed, but may use `@bitCast` for same-size conversions ```zig c.SDL_Function(@bitCast(my_u32)) ``` 5. **Return values:** - Opaque pointers: `@ptrCast` the result - Enums: `@enumFromInt` the result - Flags: `@bitCast` the result - Primitives: direct return ## Module Dependencies & Header Relationships SDL3 headers have dependencies on each other: ``` SDL_stdinc.h # Base types (Uint32, Sint32, etc.) ↓ SDL_error.h # Error handling ↓ SDL_properties.h # Properties system ↓ SDL_video.h # Video/window system ↓ SDL_gpu.h # GPU rendering (depends on video for SDL_Window) ``` **Parsing Strategy:** 1. Parse all headers into a unified AST first 2. Build type dependency graph 3. Resolve cross-header type references 4. Generate modules in dependency order 5. Add imports between generated modules as needed **Generated Module Structure:** ```zig // gpu.zig pub const c = @import("c.zig").c; pub const video = @import("video.zig"); // If needed // Use video types pub const Window = video.Window; ``` ## Simplified Zig Parser Architecture ### Key Insight: SDL3 Headers Are EXTREMELY Regular After analyzing the actual SDL3 headers, they follow **very simple patterns**: 1. **Opaque types:** `typedef struct SDL_Foo SDL_Foo;` - Single line! 2. **Enums:** `typedef enum SDL_Foo { ... } SDL_Foo;` - Braces are balanced 3. **Structs:** `typedef struct SDL_Foo { ... } SDL_Foo;` - Same as enums 4. **Flags:** `typedef Uint32 SDL_FooFlags;` + `#define SDL_FOO_*` lines following 5. **Functions:** `extern SDL_DECLSPEC Type SDLCALL SDL_Name(...);` - May span lines **We don't need:** - ❌ Full lexer/tokenizer - ❌ Recursive descent parser - ❌ Abstract Syntax Tree - ❌ Symbol tables - ❌ Type resolution - ❌ Semantic analysis - ❌ Following #includes or system headers - ❌ Complex preprocessor **We DO need:** - ✅ Line-by-line reader with brace tracking - ✅ Simple pattern matching (regex or string matching) - ✅ Extract pattern data into structs - ✅ Direct code generation ### Simplified Pipeline ``` C Header File ↓ [1. Pattern Scanner] → Extract Declarations ↓ (opaque, enum, struct, flags, function) [2. Data Extraction] → Simple Structs ↓ (name, fields, values, etc.) [3. Code Generator] → Zig Source Code ↓ Generated Bindings ``` **Complexity Reduction:** ~1/5th the original complexity! ### Simplified Module Structure #### `parser.zig` - Main Entry Point & Scanner The main file that does pattern scanning and code generation. ```zig const std = @import("std"); const patterns = @import("patterns.zig"); const codegen = @import("codegen.zig"); pub fn main() !void { var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator); defer arena.deinit(); const allocator = arena.allocator(); // 1. Parse command line arguments const args = try std.process.argsAlloc(allocator); // 2. Read header file const source = try std.fs.cwd().readFileAlloc(allocator, header_path, 10 * 1024 * 1024); // 3. Scan for patterns const declarations = try patterns.scan(allocator, source); // 4. Generate Zig code const output = try codegen.generate(allocator, declarations); // 5. Write output try std.fs.cwd().writeFile(output_path, output); } ``` **Responsibilities:** - Command-line argument parsing - File I/O - Call scanner and generator - Memory management (arena allocator) #### `patterns.zig` - Pattern Scanner Scans the C header and extracts declarations using simple pattern matching. ```zig pub const Declaration = union(enum) { opaque_type: OpaqueType, enum_decl: EnumDecl, struct_decl: StructDecl, flag_decl: FlagDecl, function_decl: FunctionDecl, }; pub const OpaqueType = struct { name: []const u8, // SDL_GPUDevice doc_comment: ?[]const u8, // /** ... */ }; pub const EnumDecl = struct { name: []const u8, // SDL_GPUPrimitiveType values: []EnumValue, // List of enum values doc_comment: ?[]const u8, }; pub const EnumValue = struct { name: []const u8, // SDL_GPU_PRIMITIVETYPE_TRIANGLELIST value: ?[]const u8, // Optional explicit value comment: ?[]const u8, // Inline comment }; pub const StructDecl = struct { name: []const u8, // SDL_GPUViewport fields: []FieldDecl, doc_comment: ?[]const u8, }; pub const FieldDecl = struct { name: []const u8, // x type_name: []const u8, // float comment: ?[]const u8, }; pub const FlagDecl = struct { name: []const u8, // SDL_GPUTextureUsageFlags underlying_type: []const u8, // Uint32 flags: []FlagValue, doc_comment: ?[]const u8, }; pub const FlagValue = struct { name: []const u8, // SDL_GPU_TEXTUREUSAGE_SAMPLER value: []const u8, // (1u << 0) comment: ?[]const u8, }; pub const FunctionDecl = struct { name: []const u8, // SDL_CreateGPUDevice return_type: []const u8, // SDL_GPUDevice * params: []ParamDecl, doc_comment: ?[]const u8, }; pub const ParamDecl = struct { name: []const u8, // format_flags type_name: []const u8, // SDL_GPUShaderFormat }; pub const Scanner = struct { source: []const u8, pos: usize, pub fn init(source: []const u8) Scanner { return .{ .source = source, .pos = 0 }; } pub fn scan(self: *Scanner, allocator: Allocator) ![]Declaration { var decls = std.ArrayList(Declaration).init(allocator); while (!self.isAtEnd()) { if (try self.scanOpaque()) |opaque| { try decls.append(.{ .opaque_type = opaque }); } else if (try self.scanEnum()) |enum_| { try decls.append(.{ .enum_decl = enum_ }); } else if (try self.scanStruct()) |struct_| { try decls.append(.{ .struct_decl = struct_ }); } else if (try self.scanFlags()) |flags| { try decls.append(.{ .flag_decl = flags }); } else if (try self.scanFunction()) |func| { try decls.append(.{ .function_decl = func }); } else { self.skipLine(); } } return decls.toOwnedSlice(); } fn scanOpaque(self: *Scanner) !?OpaqueType { // Look for: typedef struct SDL_Foo SDL_Foo; if (self.matchLine("typedef struct ")) { // Extract name from "SDL_Foo SDL_Foo;" // ... } return null; } fn scanEnum(self: *Scanner) !?EnumDecl { // Look for: typedef enum SDL_Foo // Then collect until } SDL_Foo; // ... } fn scanStruct(self: *Scanner) !?StructDecl { // Same as enum but for structs // ... } fn scanFlags(self: *Scanner) !?FlagDecl { // Look for: typedef Uint32 SDL_FooFlags; // Then collect following #define lines // ... } fn scanFunction(self: *Scanner) !?FunctionDecl { // Look for: extern SDL_DECLSPEC Type SDLCALL SDL_Name(...); // May span multiple lines // ... } // Utility functions fn matchLine(self: *Scanner, prefix: []const u8) bool { } fn readUntil(self: *Scanner, terminator: u8) []const u8 { } fn readBraced(self: *Scanner) []const u8 { } // Read {...} fn extractDocComment(self: *Scanner) ?[]const u8 { } fn skipLine(self: *Scanner) void { } fn isAtEnd(self: *Scanner) bool { } }; ``` **Strategy:** - Simple line-by-line scanning - Pattern matching with `std.mem.startsWith` - Brace counting for `{...}` blocks - Store raw strings, parse during generation #### `naming.zig` - Name Conversion Simple string manipulation for name conversion. ```zig pub fn stripPrefix(name: []const u8, prefix: []const u8) []const u8 { if (std.mem.startsWith(u8, name, prefix)) { return name[prefix.len..]; } return name; } pub fn typeNameToZig(c_name: []const u8) []const u8 { // SDL_GPUDevice -> GPUDevice (just strip SDL_) return stripPrefix(c_name, "SDL_"); } pub fn functionNameToZig(c_name: []const u8, allocator: Allocator) ![]const u8 { // SDL_CreateGPUDevice -> createGPUDevice const without_prefix = stripPrefix(c_name, "SDL_"); return lowerFirstChar(without_prefix, allocator); } pub fn enumValueToZig(c_name: []const u8, prefix: []const u8, allocator: Allocator) ![]const u8 { // SDL_GPU_PRIMITIVETYPE_TRIANGLELIST -> primitivetypeTrianglelist const without_prefix = stripPrefix(c_name, prefix); return toLowerCamelCase(without_prefix, allocator); } pub fn detectCommonPrefix(names: []const []const u8) []const u8 { // Find longest common prefix // SDL_GPU_PRIMITIVETYPE_TRIANGLELIST, SDL_GPU_PRIMITIVETYPE_LINESTRIP // -> SDL_GPU_PRIMITIVETYPE_ } fn lowerFirstChar(s: []const u8, allocator: Allocator) ![]const u8 { var result = try allocator.dupe(u8, s); if (result.len > 0) result[0] = std.ascii.toLower(result[0]); return result; } fn toLowerCamelCase(s: []const u8, allocator: Allocator) ![]const u8 { // Convert SCREAMING_SNAKE to lowerCamelCase // Handle SDL3's conventions } ``` **Convention Rules:** | C Pattern | Zig Pattern | Example | |-----------|-------------|---------| | `SDL_FooBar` (type) | `FooBar` | `SDL_GPUDevice` → `GPUDevice` | | `SDL_FooBar` (function) | `fooBar` | `SDL_CreateGPUDevice` → `createGPUDevice` | | `SDL_FOO_BAR_BAZ` (enum) | `fooBarBaz` | `SDL_GPU_PRIMITIVETYPE_TRIANGLELIST` → `primitivetypeTrianglelist` | | `SDL_FOO_BAR` (flag) | `fooBar` | `SDL_GPU_TEXTUREUSAGE_SAMPLER` → `textureusageSampler` | #### `types.zig` - Type Conversion Simple string-based type conversion (no need to parse types fully). ```zig pub fn convertType(c_type: []const u8) []const u8 { // Simple table lookup and string replacement if (std.mem.eql(u8, c_type, "void")) return "void"; if (std.mem.eql(u8, c_type, "bool")) return "bool"; if (std.mem.eql(u8, c_type, "SDL_bool")) return "bool"; if (std.mem.eql(u8, c_type, "float")) return "f32"; if (std.mem.eql(u8, c_type, "double")) return "f64"; if (std.mem.eql(u8, c_type, "char")) return "u8"; if (std.mem.eql(u8, c_type, "int")) return "c_int"; if (std.mem.eql(u8, c_type, "Uint8")) return "u8"; if (std.mem.eql(u8, c_type, "Uint16")) return "u16"; if (std.mem.eql(u8, c_type, "Uint32")) return "u32"; if (std.mem.eql(u8, c_type, "Uint64")) return "u64"; if (std.mem.eql(u8, c_type, "Sint8")) return "i8"; if (std.mem.eql(u8, c_type, "Sint16")) return "i16"; if (std.mem.eql(u8, c_type, "Sint32")) return "i32"; if (std.mem.eql(u8, c_type, "Sint64")) return "i64"; if (std.mem.eql(u8, c_type, "size_t")) return "usize"; // Pointers - simple pattern matching if (std.mem.eql(u8, c_type, "const char *")) return "[*c]const u8"; if (std.mem.eql(u8, c_type, "void *")) return "?*anyopaque"; // SDL types - just strip SDL_ prefix if (std.mem.startsWith(u8, c_type, "SDL_")) { // SDL_GPUDevice * -> *GPUDevice // SDL_GPUTextureFormat -> GPUTextureFormat // Handle pointers and const } return c_type; // fallback } ``` **Strategy:** - Table lookup for primitives - Pattern matching for pointers - String replacement for SDL types - No need to fully parse - SDL types are very regular! #### `codegen.zig` - Code Generation Direct code generation from extracted declarations. ```zig pub const CodeGen = struct { decls: []Declaration, allocator: Allocator, output: std.ArrayList(u8), pub fn generate(allocator: Allocator, decls: []Declaration) ![]const u8 { var gen = CodeGen{ .decls = decls, .allocator = allocator, .output = std.ArrayList(u8).init(allocator), }; try gen.writeHeader(); // Generate each declaration for (decls) |decl| { switch (decl) { .opaque_type => |opaque| try gen.writeOpaque(opaque), .enum_decl => |enum_| try gen.writeEnum(enum_), .struct_decl => |struct_| try gen.writeStruct(struct_), .flag_decl => |flags| try gen.writeFlags(flags), .function_decl => |func| try gen.writeFunction(func), } } return gen.output.toOwnedSlice(); } fn writeHeader(self: *CodeGen) !void { try self.output.appendSlice("pub const c = @import(\"c.zig\").c;\n\n"); } fn writeOpaque(self: *CodeGen, opaque: OpaqueType) !void { // pub const GPUDevice = opaque {}; try self.output.writer().print("pub const {s} = opaque {{}};\n\n", .{ naming.typeNameToZig(opaque.name), }); } fn writeEnum(self: *CodeGen, enum_: EnumDecl) !void { const zig_name = naming.typeNameToZig(enum_.name); try self.output.writer().print("pub const {s} = enum(c_int) {{\n", .{zig_name}); const prefix = naming.detectCommonPrefix(/* enum values */); for (enum_.values) |value| { const zig_value = try naming.enumValueToZig(value.name, prefix, self.allocator); if (value.comment) |comment| { try self.output.writer().print(" {s}, // {s}\n", .{ zig_value, comment }); } else { try self.output.writer().print(" {s},\n", .{zig_value}); } } try self.output.appendSlice("};\n\n"); } fn writeStruct(self: *CodeGen, struct_: StructDecl) !void { const zig_name = naming.typeNameToZig(struct_.name); try self.output.writer().print("pub const {s} = extern struct {{\n", .{zig_name}); for (struct_.fields) |field| { const zig_type = types.convertType(field.type_name); if (field.comment) |comment| { try self.output.writer().print(" {s}: {s}, // {s}\n", .{ field.name, zig_type, comment, }); } else { try self.output.writer().print(" {s}: {s},\n", .{ field.name, zig_type }); } } try self.output.appendSlice("};\n\n"); } fn writeFlags(self: *CodeGen, flags: FlagDecl) !void { // pub const GPUTextureUsageFlags = packed struct(u32) { // textureusageSampler: bool = false, // ... // }; // Calculate padding, generate fields } fn writeFunction(self: *CodeGen, func: FunctionDecl) !void { // Determine if it's a method or free function const is_method = isMethod(func); if (is_method) { // Will be added to opaque type later (need second pass) } else { // Free function const zig_name = try naming.functionNameToZig(func.name, self.allocator); // Generate: pub inline fn createGPUDevice(...) ... { c.SDL_CreateGPUDevice(...); } } } }; fn isMethod(func: FunctionDecl) bool { // Check if first parameter is an opaque type if (func.params.len > 0) { const first_param_type = func.params[0].type_name; // Check if it's one of the opaque types return std.mem.startsWith(u8, first_param_type, "SDL_GPU") and std.mem.endsWith(u8, first_param_type, " *"); } return false; } ``` **Strategy:** - Direct string generation (no templates needed!) - Two passes: types first, then group methods with opaque types - Simple `std.fmt.format` for code generation - No complex AST traversal **Note:** Config can be added later if needed, but start without it for simplicity. ### Memory Management Arena allocation for simplicity: ```zig pub fn main() !void { var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator); defer arena.deinit(); const allocator = arena.allocator(); // Everything freed at once when done } ``` ### Testing Simple integration tests: ```zig test "scan opaque typedef" { const source = "typedef struct SDL_GPUDevice SDL_GPUDevice;"; var scanner = patterns.Scanner.init(source); const decls = try scanner.scan(std.testing.allocator); try std.testing.expectEqual(@as(usize, 1), decls.len); try std.testing.expect(decls[0] == .opaque_type); } test "generate enum" { const enum_decl = EnumDecl{ .name = "SDL_GPUPrimitiveType", .values = &[_]EnumValue{ .{ .name = "SDL_GPU_PRIMITIVETYPE_TRIANGLELIST", .value = null, .comment = null }, }, .doc_comment = null, }; const output = try codegen.generateEnum(enum_decl, std.testing.allocator); // Check output matches expected Zig code } ``` ### Performance **Expected:** - Parse all 85 headers: < 1 second - Memory: < 50 MB - Single-threaded (sufficient for this workload) ## Parser Architecture ### Phase 1: Lexical Analysis & Preprocessing **Input:** Raw C header files **Output:** Token stream **Tasks:** 1. Remove `SDL_begin_code.h` / `SDL_close_code.h` includes (these are preprocessor magic) 2. Strip out platform-specific `#ifdef` blocks (or handle multiple platform variants) 3. Expand or track `#define` macros (especially for flag values) 4. Tokenize the remaining C code 5. Handle multi-line comments and documentation blocks **Challenges:** - C preprocessor complexity - Platform-specific code paths - Macro expansion for flag definitions **Approach:** - Use a simple regex-based preprocessor for well-known patterns - Or use libclang Python bindings for robust parsing - Focus on public API headers only (skip internal `_c.h` files) ### Phase 2: Syntax Analysis & AST Building **Input:** Token stream **Output:** Abstract Syntax Tree (AST) **AST Node Types:** - `OpaqueType` - opaque struct typedefs - `Enum` - enum definitions with values - `FlagType` - flag typedef + associated defines - `Struct` - struct definitions - `Function` - function declarations - `Comment` - documentation blocks **Key Information to Extract:** For each type/function: - Full C name (e.g., `SDL_GPUDevice`) - Zig name (e.g., `GPUDevice`) - Documentation comment - Source location (file, line number) - Related items (`\sa` references) - Version info (`\since`) For functions: - Return type - Parameter names and types - Which opaque type it belongs to (if any) - Const/pointer qualifiers For enums: - Each enumerant name and value - Inline comments for each value For flags: - Each flag name and bit position - Backing integer type For structs: - Each field name and type - Inline comments for each field - Padding requirements ### Phase 3: Semantic Analysis **Input:** Raw AST **Output:** Enriched AST with relationships **Tasks:** 1. **Type Resolution:** - Resolve all type references to their definitions - Handle forward declarations - Build type dependency graph 2. **Function Classification:** - Identify which functions are methods vs. free functions - Group methods by opaque type - Detect constructor/destructor patterns 3. **Documentation Processing:** - Parse Doxygen tags (`\param`, `\returns`, `\sa`, `\since`) - Build cross-reference map - Extract and clean inline comments 4. **Naming Convention Application:** - Convert SDL names to Zig names - Detect and handle naming collisions - Generate consistent camelCase names 5. **Module Organization:** - Determine which Zig file each definition belongs to - Based on C header name (e.g., `SDL_gpu.h` → `gpu.zig`) - Handle cross-module dependencies ### Phase 4: Code Generation **Input:** Enriched AST **Output:** Zig source files **Generation Strategy:** 1. **Header:** ```zig pub const c = @import("c.zig").c; pub const PropertiesID = u32; // Other common imports/aliases ``` 2. **Type Definitions (Order matters!):** - First: Flag types (no dependencies) - Second: Enums (no dependencies) - Third: Opaque types (empty declarations) - Fourth: Structs (may reference above types) 3. **Free Functions:** - After all types - Grouped by category 4. **Opaque Type Methods:** - Fill in method definitions in opaque types - Maintain consistent ordering **Code Generation Templates:** For each AST node type, we need a template. Examples: **Enum Template:** ```zig pub const {ZigName} = enum(c_int) { {for each value} {zigValueName}, //{inline comment} {end for} }; ``` **Flag Template:** ```zig pub const {ZigName} = packed struct({backingType}) { {for each flag} {zigFlagName}: bool = false, {end for} {padding fields} rsvd: bool = false, }; ``` **Struct Template:** ```zig pub const {ZigName} = extern struct { {for each field} {fieldName}: {zigType}, // {inline comment} {end for} }; ``` **Free Function Template:** ```zig // {C function name} pub inline fn {zigFuncName}({params}) {returnType} { {function body with casts} } ``` **Method Template:** ```zig // {C function name} pub inline fn {zigMethodName}({params}) {returnType} { c.{cFuncName}({casts and calls}); } ``` ### Phase 5: Validation & Testing **Input:** Generated Zig files **Output:** Validated, compilable bindings **Validation Steps:** 1. **Compilation Test:** - Run `zig build` on generated files - Ensure no syntax errors - Check type correctness 2. **API Completeness:** - Compare generated API surface with C headers - Ensure no functions/types are missing - Check for extra/duplicate definitions 3. **Comparison with Hand-Written:** - Diff generated `gpu.zig` with existing `src/gpu.zig` - Verify naming conventions match - Check structure and organization 4. **Cross-Reference Validation:** - Verify all type references are resolvable - Check method ownership is correct - Ensure no circular dependencies 5. **Documentation Check:** - Verify comments are preserved - Check for formatting issues - Validate cross-references ## Recommended Implementation Milestones **Current Status:** ✅ Hello world implemented (`parser.zig` lists all 85 headers) ### Milestone 1: Pattern Scanner (2-3 days) **Goal:** Extract declarations from SDL_gpu.h **Scope:** 1. Implement `patterns.zig` with `Scanner` struct 2. Scan for opaque typedefs (simple one-line pattern) 3. Scan for enums (track braces) 4. Scan for structs (track braces) 5. Store declarations in simple structs **Deliverable:** - `patterns.zig` that extracts opaque, enum, and struct from SDL_gpu.h - Basic tests for each pattern type **Complexity:** Low - just string matching and brace counting ### Milestone 2: Code Generation (2-3 days) **Goal:** Generate Zig code for extracted declarations **Scope:** 1. Implement `codegen.zig` 2. Implement `naming.zig` for name conversion 3. Implement `types.zig` for type conversion 4. Generate opaque types 5. Generate enums 6. Generate structs **Deliverable:** - Generated Zig code for subset of SDL_gpu.h - Code compiles with `zig build` - Matches hand-written style **Complexity:** Low - direct string generation ### Milestone 3: Flags and Functions (2-3 days) **Goal:** Complete SDL_gpu.h parsing **Scope:** 1. Add flag scanning (typedef + #define lines) 2. Add function scanning 3. Classify functions (method vs. free function) 4. Generate flag types 5. Generate functions and methods **Deliverable:** - Complete `gpu.zig` generation - All types and functions included - Compiles and matches hand-written version **Complexity:** Medium - function classification logic ### Milestone 4: Multi-Header Support (1-2 days) **Goal:** Generalize to other headers **Scope:** 1. Test on SDL_video.h, SDL_events.h, SDL_init.h 2. Handle any new patterns 3. Fix bugs 4. Add integration tests **Deliverable:** - Parser handles all common SDL3 patterns - Generate bindings for multiple headers - All generated code compiles **Complexity:** Low - SDL headers are very consistent ### Total Time Estimate **2-3 weeks** of focused work (vs. 6-8 weeks with complex architecture) **Key Simplifications:** - No lexer/tokenizer (line-by-line scanning) - No AST (direct data extraction) - No semantic analysis (simple pattern matching) - No complex type system (string conversion) ## Implementation Plan ### Stage 1: Prototype Parser (Week 1-2) **Goal:** Parse SDL_gpu.h and generate gpu.zig **Tasks:** 1. Choose parsing approach (libclang vs. custom parser) 2. Implement basic token scanner 3. Parse enum definitions 4. Parse flag definitions 5. Parse struct definitions 6. Parse opaque types 7. Parse function signatures **Deliverable:** Working parser for SDL_gpu.h ### Stage 2: Code Generator (Week 2-3) **Goal:** Generate gpu.zig from parsed data **Tasks:** 1. Implement naming convention rules 2. Build type dependency resolver 3. Create code generation templates 4. Implement function classification 5. Add method grouping logic 6. Generate initial gpu.zig **Deliverable:** Generated gpu.zig that compiles ### Stage 3: Refinement (Week 3-4) **Goal:** Match hand-written gpu.zig quality **Tasks:** 1. Compare generated vs. hand-written 2. Fix naming mismatches 3. Improve comment formatting 4. Adjust code organization 5. Handle edge cases 6. Add manual override system for special cases **Deliverable:** Generated gpu.zig identical to hand-written version ### Stage 4: Generalization (Week 4-6) **Goal:** Parse all 85 SDL3 headers **Tasks:** 1. Test parser on other headers (video, events, init, etc.) 2. Handle new patterns not seen in gpu.h 3. Implement cross-header type resolution 4. Add module dependency management 5. Handle platform-specific code 6. Create configuration system for header selection **Deliverable:** Parser that handles all SDL3 headers ### Stage 5: Integration & Automation (Week 6-7) **Goal:** Integrate into build system **Tasks:** 1. Create Zig build step for code generation 2. Add header change detection 3. Implement incremental regeneration 4. Add validation step to build 5. Create documentation generator 6. Write user guide **Deliverable:** Automated, maintainable system ## Technical Decisions ### Parser Implementation **Option A: libclang bindings (C or Zig)** - ✅ Robust, handles all C syntax - ✅ Proper preprocessor support - ✅ Battle-tested - ❌ External dependency - ❌ Slower - ❌ Overkill for regular headers - ❌ Complex API **Option B: Custom Zig parser** - ✅ Lightweight and fast - ✅ Tailored to SDL patterns - ✅ Easy to debug and modify - ✅ No external dependencies - ✅ Compiles to single binary - ✅ Same language as target (Zig → Zig) - ✅ Can share types with generated code - ✅ Strong type safety during parsing - ❌ Need to handle C syntax edge cases - ❌ Manual preprocessor handling **Decision:** **Option B** (custom Zig parser). **Rationale:** SDL3 headers are extremely regular. A custom Zig parser lets us exploit this regularity and generate better Zig code. We can handle the preprocessor with simple pattern matching for the common cases. Using Zig gives us strong typing, safety, and performance, and results in a single-binary tool with no external dependencies. ### Language Choice **Zig** - Perfect for this task: - Strong type system helps model C syntax accurately - Excellent string processing with `std.mem` - Arena allocators simplify AST memory management - Fast compilation and execution - Same language as output (Zig → Zig) - Can share common types between parser and generated code - Single binary deployment - No runtime dependencies ### File Organization ``` lib/sdl3/ ├── parser/ # Simple parser implementation (Zig) │ ├── build.zig # Parser build script │ ├── parser.zig # Main entry point & CLI │ ├── patterns.zig # Pattern scanner (core logic) │ ├── codegen.zig # Code generator │ ├── naming.zig # Name conversion utilities │ └── types.zig # Type conversion utilities ├── src/ # Hand-written & final bindings (target) │ ├── gpu.zig # Reference implementation (1,198 lines) │ ├── video.zig │ ├── events.zig │ └── ... ├── SDL/ # SDL3 submodule │ └── include/SDL3/ # Source C headers (85 files) └── research/ └── sdl-header-parser.md # This file ``` **Total Modules:** 4 (vs. 10+ in over-engineered approach) **Total Complexity:** ~1/5th of original plan ### Configuration System Support manual overrides for edge cases: **config.py:** ```python # Functions that should be free functions despite taking opaque pointer first FREE_FUNCTIONS = [ 'SDL_SomeSpecialCase', ] # Custom type mappings TYPE_OVERRIDES = { 'SDL_bool': 'bool', 'void*': '*anyopaque', } # Headers to skip SKIP_HEADERS = [ 'SDL_test_*.h', # Test framework 'SDL_oldnames.h', # Deprecated ] # Custom naming rules NAMING_OVERRIDES = { 'SDL_bool': 'bool', } ``` ## Parsing Challenges & Solutions ### Challenge 1: Multi-line Declarations C allows declarations to span multiple lines: ```c extern SDL_DECLSPEC SDL_GPUDevice * SDLCALL SDL_CreateGPUDevice( SDL_GPUShaderFormat format_flags, bool debug_mode, const char *name); ``` **Solution:** Normalize whitespace before parsing, treat newlines as spaces inside declarations. ### Challenge 2: Documentation Comment Association Comments must be correctly associated with the following declaration: ```c /** * Creates a device. */ typedef struct SDL_GPUDevice SDL_GPUDevice; // This gets the comment ``` **Solution:** Track the "pending comment" and attach it to the next declaration. ### Challenge 3: Macro Values Flag values use macros: ```c #define SDL_GPU_TEXTUREUSAGE_SAMPLER (1u << 0) ``` **Solution:** Evaluate simple expressions (bit shifts, arithmetic) during parsing. ### Challenge 4: Nested Structs SDL rarely uses these, but they can appear: ```c typedef struct SDL_Foo { struct { int x, y; } point; } SDL_Foo; ``` **Solution:** Flatten or generate anonymous struct types as needed. ### Challenge 5: Function Pointers in Structs ```c typedef struct SDL_Foo { void (*callback)(void *userdata); } SDL_Foo; ``` **Solution:** Convert to Zig function pointer syntax: ```zig callback: ?*const fn (userdata: ?*anyopaque) callconv(.C) void, ``` ### Challenge 6: Forward Declarations ```c typedef struct SDL_Surface SDL_Surface; // Forward declaration // ... later ... typedef struct SDL_Surface { // actual definition } SDL_Surface; ``` **Solution:** Track forward declarations, replace with full definition when found. ### Challenge 7: Conditional Compilation ```c #ifdef SDL_PLATFORM_WIN32 typedef HWND SDL_WindowHandle; #else typedef void* SDL_WindowHandle; #endif ``` **Solution:** Either: - Parse all branches and generate conditional Zig code - Use platform-specific configuration - Default to most general case ## Edge Cases & Special Handling ### 1. Properties API SDL3 has a properties system with string constants: ```c #define SDL_PROP_GPU_DEVICE_CREATE_DEBUGMODE_BOOLEAN "SDL.gpu.device.create.debugmode" ``` These should be preserved as string constants in Zig. ### 2. Callbacks Function pointer types need special handling: ```c typedef void (*SDL_SomeCallback)(void *userdata); ``` Map to Zig function pointers: ```zig pub const SomeCallback = *const fn (userdata: ?*anyopaque) callconv(.C) void; ``` ### 3. Union Types SDL uses unions in some places: ```c typedef union SDL_Event { Uint32 type; SDL_WindowEvent window; } SDL_Event; ``` Map to Zig extern unions: ```zig pub const Event = extern union { type: u32, window: WindowEvent, }; ``` ### 4. Variadic Functions Some SDL functions are variadic (e.g., `SDL_Log`). These should be marked appropriately or wrapped. ### 5. Platform-Specific Types Handle with conditional compilation: ```zig pub const WindowsHandle = if (builtin.os.tag == .windows) *c.HWND else *anyopaque; ``` ### 6. Anonymous Structs/Enums These rarely appear in SDL3 public headers but should be handled if encountered. ## Success Criteria The parser/generator is successful when: 1. ✅ All 85 SDL3 headers can be parsed without errors 2. ✅ Generated Zig code compiles without warnings 3. ✅ Generated API is 100% complete (no missing functions/types) 4. ✅ Generated code matches hand-written style 5. ✅ Documentation is preserved and readable 6. ✅ Build time is reasonable (<5 seconds for full regeneration) 7. ✅ Integration tests pass with generated bindings 8. ✅ Code is maintainable and well-documented ## Future Enhancements ### Phase 2 Features 1. **Multi-language support:** Generate bindings for other languages 2. **Documentation generation:** Create API documentation from parsed data 3. **Test generation:** Auto-generate basic API tests 4. **Type-safe wrappers:** Generate higher-level Zig wrappers with better error handling 5. **Backwards compatibility:** Handle multiple SDL versions ## Open Questions 1. **Preprocessor handling:** How much preprocessor complexity do we need to support? - **Answer:** Start simple, expand as needed 2. **Manual overrides:** How do we handle cases where generated code isn't quite right? - **Answer:** Configuration file + ability to exclude certain items from generation 3. **Version tracking:** How do we track which SDL version we're generating for? - **Answer:** Parse version from SDL_version.h, embed in generated files 4. **Breaking changes:** What happens when SDL API changes? - **Answer:** Regenerate, review diff, update override config if needed 5. **Testing strategy:** How do we test the generated bindings? - **Answer:** Compile tests + comparison with hand-written + integration tests ## Example Workflow Here's how the parser would be used in practice: ```bash # Build the parser $ cd lib/sdl3/parser $ zig build # Test it lists headers correctly $ zig build run -- ../SDL/include/SDL3 SDL3 Header Parser ================== Scanning headers in: ../SDL/include/SDL3 [1] SDL_gpu.h [2] SDL_video.h ... Total headers found: 85 # Parse a single header (future) $ zig build run -- ../SDL/include/SDL3/SDL_gpu.h --output ../src/gpu.zig # Parse all headers (future) $ zig build run -- ../SDL/include/SDL3 --output-dir ../src # Compare with hand-written $ diff ../src/gpu.zig ../src/gpu.zig.backup # Build and test the generated bindings $ cd ../.. && zig build test ``` **Recommended Development Workflow:** 1. **Implement lexer** with comprehensive tests 2. **Implement syntax parser** for basic patterns (enum, struct, function) 3. **Implement code generator** for those patterns 4. **Test on subset of SDL_gpu.h** (see "Recommended First Milestone") 5. **Iterate until output matches** hand-written bindings 6. **Add semantic analysis** (type resolution, function classification) 7. **Extend to full SDL_gpu.h** 8. **Generalize to other headers** one by one 9. **Add config system** for edge cases 10. **Integrate into build system** for automatic regeneration ## Troubleshooting Guide ### Problem: Generated code doesn't compile **Possible Causes:** 1. Type conversion is wrong (check C type → Zig type mapping) 2. Cast is missing or incorrect (check @ptrCast, @bitCast usage) 3. Missing import (check module dependencies) 4. Struct field alignment issue (use `extern struct`) **Solution:** - Compare with hand-written version - Check `zig build` error message carefully - Verify the C type in header matches assumption ### Problem: Parser fails to extract a declaration **Possible Causes:** 1. Multi-line declaration not handled 2. Unexpected syntax/formatting 3. Preprocessor directive interfering 4. Comment breaking parser **Solution:** - Print the problematic line with context - Check for unusual formatting - Simplify the declaration in a test case - Add special handling for this pattern ### Problem: Generated function doesn't match hand-written **Possible Causes:** 1. Function classification is wrong (method vs. free function) 2. Parameter types differ 3. Cast strategy differs 4. Naming convention mismatch **Solution:** - Review function classification rules - Check parameter type conversions - Verify cast strategy for each type - Update naming rules in config ### Problem: Flag structure has wrong padding **Possible Causes:** 1. Bit positions calculated incorrectly 2. Backing type size wrong (u32 vs u64) 3. Missing flags **Solution:** - Verify all flag definitions are found - Check bit position extraction - Ensure padding calculation accounts for all bits - Validate backing type matches typedef ### Problem: Cross-reference types not found **Possible Causes:** 1. Type defined in different header 2. Forward declaration not resolved 3. Module dependency missing **Solution:** - Parse dependent headers first - Build complete type database - Add explicit imports in generated code - Check type dependency graph ## References - SDL3 repository: https://github.com/libsdl-org/SDL - SDL3 headers: `lib/sdl3/SDL/include/SDL3/` - Existing bindings: `lib/sdl3/src/` - Zig documentation: https://ziglang.org/documentation/master/ - libclang Python: https://libclang.readthedocs.io/ - C11 Standard: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf - Zig Language Reference: https://ziglang.org/documentation/master/ ## Next Steps **Current Status:** ✅ Hello world implemented ### Immediate Next Steps (This Week) 1. **Implement `patterns.zig`** (2-3 days) - Create simple scanner that reads line-by-line - Match pattern: `typedef struct SDL_Foo SDL_Foo;` → opaque - Match pattern: `typedef enum SDL_Foo {` ... `} SDL_Foo;` → enum - Match pattern: `typedef struct SDL_Foo {` ... `} SDL_Foo;` → struct - Store in simple structs (no complex AST) 2. **Implement `codegen.zig`** + helpers (2-3 days) - Create `naming.zig` for name conversion (SDL_GPUDevice → GPUDevice) - Create `types.zig` for type conversion (Uint32 → u32, float → f32) - Generate Zig code directly from extracted data - Test on subset of SDL_gpu.h 3. **Add flags and functions** (2-3 days) - Parse flag typedefs + #define sequences - Parse function declarations - Classify as method or free function (check first param) - Generate complete gpu.zig ### Following Week 4. **Test on other headers** (1-2 days) - Try SDL_video.h, SDL_events.h - Fix any new patterns - Handle edge cases 5. **Polish and integrate** (1-2 days) - Clean up code - Add tests - Update build system - Document usage **Total: 2-3 weeks** to complete parser ## Key Takeaways 1. **SDL3 headers are EXTREMELY regular** - Perfect for simple pattern matching 2. **Don't over-engineer** - Text transformation is sufficient, no need for full parser 3. **Start small** - Get pattern matching working for one header first 4. **Use hand-written as reference** - The existing gpu.zig shows exactly what we want 5. **Iterate quickly** - Scan, generate, compile, compare, fix, repeat 6. **Line-by-line scanning works** - No need for tokenizer/lexer 7. **Direct generation is simpler** - No need for AST, just extract and generate 8. **Simple pattern matching** - `typedef struct SDL_Foo SDL_Foo;` is a one-line pattern 9. **Brace counting is enough** - Track `{` and `}` for multi-line declarations 10. **String conversion for types** - Table lookup, no need to parse type expressions 11. **Function classification is simple** - Check if first param is opaque type 12. **Implementation time: 2-3 weeks** (vs. 6-8 weeks for over-engineered approach) ## Conclusion This plan provides a comprehensive roadmap for creating an SDL3 header parser and Zig binding generator. The regular structure of SDL3 headers makes this an ideal project for automated code generation. By following this plan, we can create maintainable, high-quality Zig bindings that stay synchronized with SDL3 development. The project is feasible because: - **SDL3 headers are EXTREMELY regular** - Simple pattern matching works - **We have excellent reference implementations** - Hand-written bindings show target output - **Text transformation is sufficient** - No need for complex parsing - **The scope is well-defined** - 85 headers, 5-6 simple patterns - **Zig provides excellent tooling** - String manipulation, arena allocation, fast compilation - **No external dependencies** - Single binary, easy integration With a simplified approach using pattern matching instead of full parsing, this can be completed in **2-3 weeks** of focused work (vs. 6-8 weeks for over-engineered approach). The result will be a maintainable system that generates high-quality Zig bindings automatically. ### Advantages of Simplified Approach 1. **Simplicity:** ~500 lines of code vs. 2000+ for full parser 2. **Speed:** Faster to implement and faster to execute 3. **Maintainability:** Easy to understand and modify 4. **Reliability:** Less code = fewer bugs 5. **Sufficiency:** SDL3 headers don't need full C parsing 6. **Quick iteration:** Changes are fast to test ### Simplified Parser Flow ``` C Header File ↓ ┌──────────────────────────────────┐ │ patterns.zig (Scanner) │ │ Line-by-line pattern matching │ │ - typedef struct SDL_Foo... │ │ - typedef enum SDL_Foo {... │ │ - typedef Uint32 SDL_Flags; │ │ - extern SDL_DECLSPEC... │ └──────────────────┬───────────────┘ │ ▼ Simple Struct Data (name, fields, values, etc.) │ ▼ ┌──────────────────────────────────┐ │ codegen.zig │ │ Direct string generation │ │ + naming.zig (name conversion) │ │ + types.zig (type conversion) │ └──────────────────┬───────────────┘ │ ▼ gpu.zig (output) ``` **Complexity:** Very Low - just pattern matching and string generation!