Add parser documentation and clean up planning files

Added:
- PARSER_OVERVIEW.md: Concise guide on how the parser works
  - Architecture overview
  - Input/output examples
  - Usage instructions
  - Statistics and limitations

- AGENTS.md: Added 3 new issues from mock testing work
  - Build.addStaticLibrary removal in Zig 0.15
  - C mock type definition requirements
  - Testing strategy lessons

Removed completed planning/work log files:
- MOCK_FLAG_UPDATE.md
- PHASE1_COMPLETE.md
- SUMMARY.md
- TEST_HARNESS_PLAN_V2.md

Keeping only essential docs: AGENTS.md, PARSER_OVERVIEW.md, DEPENDENCY_PLAN.md, TODO.md
This commit is contained in:
Peterino2 2026-01-22 01:36:16 -08:00
parent fd37a11da8
commit d5b381526c
6 changed files with 185 additions and 1542 deletions

View File

@ -305,6 +305,80 @@ defer allocator.free(formatted);
- **Date**: 2025-01-22
- **SDL Version**: 3.2.0
## Issues Encountered During Mock Testing Implementation
### Issue 1: Build.addStaticLibrary Removed
**Problem**: Zig 0.15 removed `b.addStaticLibrary()` method.
**Error**:
```
error: no field or member function named 'addStaticLibrary' in 'Build'
```
**Solution**: Use `b.addLibrary()` with `.linkage = .static`:
```zig
// OLD - Does not work
const lib = b.addStaticLibrary(.{
.name = "mylib",
.target = target,
.optimize = optimize,
});
// NEW - Correct for Zig 0.15
const lib = b.addLibrary(.{
.name = "mylib",
.linkage = .static,
.root_module = b.createModule(.{
.target = target,
.optimize = optimize,
}),
});
```
**Key change**: Must create `root_module` explicitly with target/optimize.
### Issue 2: C Mock Type Definitions
**Problem**: Generated C mocks referenced SDL types like `Uint32`, `SDL_Window`, `FColor` that weren't defined when using only stdint.h/stdbool.h.
**Error**:
```
error: unknown type name 'Uint32'
error: unknown type name 'SDL_GPUColorTargetInfo'
```
**Solution**: Include actual SDL headers in generated mocks:
```c
// OLD - Missing types
#include <stdint.h>
#include <stdbool.h>
#include <stddef.h>
// NEW - Proper type definitions
#include <SDL3/SDL_stdinc.h>
#include <SDL3/SDL_gpu.h>
```
Then add SDL include path to C compilation:
```zig
mock_lib.addIncludePath(b.path("SDL/include"));
```
**Key insight**: Mocks should compile like real SDL implementation files, with full access to SDL type definitions.
### Issue 3: Testing Strategy
**Problem**: Initial testing with tiny `test_small.h` (3 declarations) didn't reveal real-world issues.
**Solution**: Test with full production header (SDL_gpu.h with 169 declarations) to:
- Verify parser handles large inputs
- Catch type definition issues
- Validate all declaration types work together
- Ensure build system scales
**Lesson**: Always test with realistic, production-sized inputs, not toy examples.
## References
- Zig 0.15 Release Notes: https://ziglang.org/download/0.15.0/release-notes.html

View File

@ -1,101 +0,0 @@
# Mock Flag Update
## Summary
Updated the `--mocks` flag to accept an explicit output path, improving usability and integration with build systems.
## Changes Made
### 1. Flag Syntax Change
**Before:**
```bash
zig build run -- header.h --output=bindings.zig --mocks
# Automatically created: header_mock.c
```
**After:**
```bash
zig build run -- header.h --output=bindings.zig --mocks=mocks.c
# Explicitly creates: mocks.c
```
### 2. Benefits
- **Explicit control**: Users specify exactly where mock file goes
- **Build system friendly**: Easy to integrate with Zig build system
- **Cleaner**: No automatic filename generation logic
- **Flexible**: Can output mocks anywhere in the project structure
### 3. Build Target Added
New `test-mocks` target for quick testing:
```bash
zig build test-mocks
# Generates: zig-out/test_small.zig and zig-out/test_small_mock.c
```
### 4. Files Modified
**build.zig**:
- Added `test-mocks` build step
- Outputs to `zig-out/` directory by default
- Uses absolute paths for consistency
**parser.zig**:
- Changed from `--mocks` (boolean flag) to `--mocks=<path>` (value flag)
- Removed automatic filename generation
- Updated usage documentation
**docs/usage.md**:
- Updated with new flag syntax
- Added command line options reference
- Added `test-mocks` target documentation
**PHASE1_COMPLETE.md**:
- Updated examples with new syntax
- Documented build system integration
## Examples
### Simple test:
```bash
zig build test-mocks
```
### Custom paths:
```bash
zig build run -- SDL_gpu.h --output=gen/bindings.zig --mocks=gen/mocks.c
```
### Just bindings (no mocks):
```bash
zig build run -- header.h --output=bindings.zig
```
## Backward Compatibility
**Breaking change**: The old `--mocks` flag (without a value) no longer works.
**Migration**:
```bash
# Old (no longer works)
zig build run -- header.h --output=out.zig --mocks
# New (required)
zig build run -- header.h --output=out.zig --mocks=header_mock.c
```
## Testing
All existing tests pass:
- ✅ 7 mock generation unit tests
- ✅ Parser tests
- ✅ Integration with test_small.h
- ✅ Integration with SDL_gpu.h (169 declarations)
- ✅ New `test-mocks` build target
## Implementation Time
- **Estimated**: 30 minutes
- **Actual**: 25 minutes
- Flag update: 10 minutes
- Build target: 10 minutes
- Documentation: 5 minutes

111
lib/sdl3/parser/PARSER_OVERVIEW.md vendored Normal file
View File

@ -0,0 +1,111 @@
# SDL3 Parser - Overview
## What It Does
Automatically generates type-safe Zig bindings and C mock implementations from SDL3 C headers.
## How It Works
### 1. Lexical Analysis (patterns.zig)
- Scans C header files for SDL API patterns
- Extracts 5 declaration types:
- **Opaque types**: `typedef struct SDL_Type SDL_Type;`
- **Enums**: `typedef enum { ... } SDL_Type;`
- **Structs**: `typedef struct { ... } SDL_Type;`
- **Flags**: Packed bitfields from enums
- **Functions**: `extern SDL_DECLSPEC RetType SDLCALL SDL_Func(...);`
### 2. Type Conversion (types.zig)
- Maps C types to Zig equivalents:
- `bool``bool`
- `Uint32``u32`
- `SDL_Type*``?*Type` (nullable) or `*Type` (non-null)
- `void*``?*anyopaque`
- `const char*``[*c]const u8`
### 3. Naming Convention (naming.zig)
- Strips `SDL_` prefix
- Removes first underscore for grouping: `SDL_GPU_Device``GPUDevice`
- Converts to camelCase: `SDL_CreateGPUDevice``createGPUDevice`
### 4. Code Generation (codegen.zig)
- **Groups methods**: Functions with matching first parameter go inside opaque type
- **Generates inline wrappers**: Handle casting between Zig and C types
- **Formats output**: Uses Zig AST for proper formatting
### 5. Mock Generation (mock_codegen.zig)
- Creates C stub implementations for testing
- Includes actual SDL headers for type definitions
- Returns null/0/false for all functions
## Example
**Input** (SDL_gpu.h):
```c
typedef struct SDL_GPUDevice SDL_GPUDevice;
extern SDL_DECLSPEC SDL_GPUDevice* SDLCALL SDL_CreateGPUDevice(bool debug);
```
**Output Zig** (gpu.zig):
```zig
pub const GPUDevice = opaque {};
pub inline fn createGPUDevice(debug: bool) ?*GPUDevice {
return c.SDL_CreateGPUDevice(debug);
}
```
**Output Mock** (gpu_mock.c):
```c
#include <SDL3/SDL_gpu.h>
SDL_GPUDevice* SDL_CreateGPUDevice(bool debug) {
(void)debug;
return NULL;
}
```
## Usage
```bash
# Generate bindings only
zig build run -- SDL_gpu.h --output=gpu.zig
# Generate bindings + mocks
zig build run -- SDL_gpu.h --output=gpu.zig --mocks=gpu_mock.c
# Test with SDL_gpu.h
zig build test-mocks
```
## Architecture
```
C Header → Scanner → AST → Type Mapper → Code Generator → Zig Bindings
Mock Generator → C Mocks
```
## Statistics (SDL_gpu.h)
- **Input**: 169 declarations
- **Output**: 1,229 lines of Zig, 577 lines of C mocks
- **Compilation**: 71KB static library, 94 exported functions
- **Tests**: 7/7 passing
## Key Features
✅ Type-safe pointer handling (nullable vs non-null)
✅ Automatic method grouping in opaque types
✅ Minimal casting (only where needed)
✅ AST-based formatting
✅ C mocks with real SDL headers
✅ Handles large headers (169+ declarations)
## Limitations
- No dependency resolution (types from other headers)
- No `#define` parsing (except simple enums)
- No function pointer types
- No union types
- Requires manual `c.zig` for imports

View File

@ -1,183 +0,0 @@
# Phase 1 Complete: Mock Code Generator
## Summary
Successfully implemented C mock code generation for the SDL3 parser using Test-Driven Development (TDD).
## Completed Features ✅
### 1. Mock Code Generator (`mock_codegen.zig`)
- **Lines of Code**: ~145 lines
- **Test Coverage**: 7 unit tests, all passing
- **Functionality**:
- Generates C header with proper includes (`stdint.h`, `stdbool.h`, `stddef.h`)
- Generates forward declarations for opaque types
- Generates stub functions with:
- Proper function signatures matching C declarations
- Parameter voiding to avoid unused warnings
- Appropriate default return values:
- `NULL` for pointer types
- `false` for bool types
- `0` for integer types
- `0.0` for float types
- No return for void functions
### 2. Parser Integration
- **Updated `parser.zig`**:
- Added `--mocks=<path>` flag support (specifies output path for mocks)
- Improved multi-flag argument parsing
- Updated usage documentation
### 3. Build System Integration
- **Updated `build.zig`**:
- Added `test-mocks` build target
- Outputs to `zig-out/` directory by default
- Usage: `zig build test-mocks`
### 4. Test Results
**Unit Tests** (mock_codegen_test.zig):
```
7/7 mock_codegen tests passed:
✅ Simple function generation
✅ Void function generation
✅ Opaque type forward declarations
✅ Header and includes
✅ Multiple parameters
✅ Bool return type
✅ Int return type
```
**Integration Test** (test_small.h):
```bash
$ zig build test-mocks
Generated: zig-out/test_small.zig
Generated C mocks: zig-out/test_small_mock.c
```
**Full SDL Test** (SDL_gpu.h):
```bash
$ zig build run -- ../SDL/include/SDL3/SDL_gpu.h --output=zig-out/SDL_gpu.zig --mocks=zig-out/SDL_gpu_mock.c
Found 169 declarations
- Opaque types: 13
- Enums: 24
- Structs: 35
- Flags: 3
- Functions: 94
Generated: zig-out/SDL_gpu.zig
Generated C mocks: zig-out/SDL_gpu_mock.c (18KB, 593 lines)
```
## Example Generated Mock
**Input** (C header):
```c
typedef struct SDL_GPUDevice SDL_GPUDevice;
extern SDL_DECLSPEC SDL_GPUDevice* SDLCALL SDL_CreateGPUDevice(bool debug_mode);
```
**Output** (C mock):
```c
// Auto-generated C mock implementations
// DO NOT EDIT - Generated by sdl-parser --mocks
#include <stdint.h>
#include <stdbool.h>
#include <stddef.h>
// Forward declarations for opaque types
typedef struct SDL_GPUDevice SDL_GPUDevice;
// Function implementations
SDL_GPUDevice* SDL_CreateGPUDevice(bool debug_mode) {
(void)debug_mode;
return NULL;
}
```
## Usage
### Using build target:
```bash
# Test with small header
zig build test-mocks
# Output: zig-out/test_small.zig and zig-out/test_small_mock.c
```
### Generate Zig bindings only:
```bash
zig build run -- header.h --output=bindings.zig
```
### Generate Zig bindings + C mocks:
```bash
zig build run -- header.h --output=bindings.zig --mocks=mocks.c
# Creates: bindings.zig and mocks.c
```
### Using stdout (legacy, Zig output only):
```bash
zig build run -- header.h > bindings.zig
```
## Next Steps (Phase 2)
According to TEST_HARNESS_PLAN_V2.md:
1. ⚠️ **Test Project Setup** (2 hours)
- Create test_project directory structure
- Write build.zig that compiles mocks and tests
- Set up integration testing
2. ⚠️ **Basic Test Runner** (2 hours)
- Implement opaque type tests
- Implement enum/struct/flag tests
- Test with generated output
3. ⚠️ **Function Coverage** (2 hours)
- Generate tests for all 94 functions
- Verify linkage works
- Handle nullable pointers
4. ⚠️ **Fix Remaining Syntax Errors** (2-4 hours)
- 59 syntax errors remain in full SDL output
- Investigate and fix edge cases
## Time Spent
- **Estimated**: 3 hours
- **Actual**: ~3 hours
- Test writing: 0.5 hours
- Implementation: 1 hour
- Integration & debugging: 1 hour
- Flag update & build integration: 0.5 hours
## Files Created/Modified
### Created:
- `mock_codegen.zig` (145 lines)
- `mock_codegen_test.zig` (185 lines)
- `PHASE1_COMPLETE.md` (this file)
### Modified:
- `parser.zig` - Changed `--mocks` to `--mocks=<path>` for explicit output path
- `build.zig` - Added `test-mocks` target
- `TEST_HARNESS_PLAN_V2.md` - Updated with Phase 0 completion status
### Generated (test outputs in zig-out/):
- `test_small_mock.c` (364 bytes)
- `test_small.zig` (291 bytes)
- `SDL_gpu_mock.c` (18KB)
- `SDL_gpu.zig` (51KB)
## Notes
- Mock files reference SDL types (like `SDL_Window`, `Uint32`) which aren't defined in the mocks themselves
- This is intentional - mocks are meant to be compiled alongside SDL headers or with type definitions
- For standalone testing, additional type definitions would be needed
- All tests use TDD approach: tests written first, implementation second
- Mock generation adds minimal overhead to parser runtime (~50ms for SDL_gpu.h)
- The `--mocks=<path>` flag provides explicit control over output location
- Output files now go to `zig-out/` by default for cleaner project structure

View File

@ -1,258 +0,0 @@
# SDL3 Parser - Work Summary
## Project Overview
A Zig-based parser that automatically generates type-safe Zig bindings from SDL3 C headers. Successfully parses SDL_gpu.h (169 declarations) and generates production-quality bindings with ergonomic method syntax.
## What Was Accomplished
### 1. Core Parser Features ✅
**Type Support:**
- ✅ Opaque types (13 in SDL_gpu.h)
- ✅ Enums (24 in SDL_gpu.h)
- ✅ Structs (35 in SDL_gpu.h)
- ✅ Flags/Bitfields (3 in SDL_gpu.h)
- ✅ Functions (94 in SDL_gpu.h)
**Advanced Type Handling:**
- ✅ Double pointers (`SDL_Type **` → `?*?*Type`)
- ✅ Const pointer arrays (`SDL_Type *const *` → `[*c]*const Type`)
- ✅ Output parameters (`Uint32 *` → `*u32`)
- ✅ Nullable vs non-nullable pointers
- ✅ Proper primitive pointer types
### 2. Code Generation Features ✅
**Method Organization:**
- ✅ Functions grouped inside opaque types as methods
- ✅ First parameter becomes `self` (e.g., `gpudevice: *GPUDevice`)
- ✅ Non-nullable pointers in method signatures
- ✅ Standalone functions for module-level APIs
**Formatting:**
- ✅ AST-based formatting (uses `std.zig.Ast.renderAlloc`)
- ✅ Smart trailing commas (only for 4+ parameters)
- ✅ Proper indentation and line breaks
- ✅ Comment preservation
**Type Safety:**
- ✅ Automatic cast insertion (`@ptrCast`, `@bitCast`, `@intFromEnum`)
- ✅ Minimal casting (no unnecessary casts for value types)
- ✅ Better types than handwritten version
### 3. Build Integration ✅
**Package Setup:**
- ✅ `build.zig.zon` with proper fingerprint
- ✅ Integrated into SDL3 build system
- ✅ `regenerate-zig` build step
- ✅ Automatic generation on demand
**Output:**
- ✅ Generates to `v2/gpu.zig`
- ✅ 1229 lines of type-safe bindings
- ✅ Zero syntax errors
- ✅ All tests passing
### 4. Zig 0.15 Compatibility ✅
**Fixed Issues:**
- ✅ ArrayList API changes (now unmanaged)
- ✅ AST rendering API changes
- ✅ Proper allocator threading
- ✅ Updated all collection operations
### 5. Documentation ✅
**Created:**
- ✅ `AGENTS.md` - Zig 0.15 solutions guide
- ✅ `SUMMARY.md` - This file
- ✅ Dependency resolution plan
- ✅ Inline code comments
## Generated API Example
```zig
// Ergonomic method syntax
pub const GPUDevice = opaque {
pub inline fn createGPUTexture(
gpudevice: *GPUDevice,
createinfo: *const GPUTextureCreateInfo,
) ?*GPUTexture {
return c.SDL_CreateGPUTexture(gpudevice, @ptrCast(createinfo));
}
};
// Usage
const texture = device.createGPUTexture(&info);
```
## Quality Metrics
| Metric | Value |
|--------|-------|
| Declarations Parsed | 169 |
| Syntax Errors | 0 |
| Type Safety | Improved over handwritten |
| Lines of Code | 1,229 |
| Test Coverage | All existing tests pass |
| Build Errors | None |
## Known Limitations
### 1. Missing Dependency Types ⚠️
Generated code references types from other SDL headers:
- `FColor` (SDL_pixels.h)
- `Rect` (SDL_rect.h)
- `PropertiesID` (SDL_properties.h)
- `Window` (SDL_video.h)
- `FlipMode` (SDL_surface.h)
- `GPUShaderFormat` (special case: #define flags)
**Status**: Implementation plan created (see below)
### 2. Not Yet Implemented
- ❌ #define-based flags parsing
- ❌ Function pointer typedefs
- ❌ Callback types
- ❌ Dependency resolution
- ❌ Multi-header generation
## Next Steps - Dependency Resolution
### Planned Implementation
**Phase 1: Dependency Detection**
- Scan generated code for non-target types
- Map types to source headers (from #include directives)
- Build minimal dependency list
**Phase 2: Selective Extraction**
- Parse dependency headers
- Extract ONLY referenced types
- Generate minimal `<module>.zig` files
**Phase 3: Integration**
- Generate imports in main file
- Handle special cases (opaque types, #defines)
- Verify compilation
### Expected File Structure
```
v2/
├── gpu.zig # Main file with imports
├── pixels.zig # FColor only
├── rect.zig # Rect only
├── properties.zig # PropertiesID only
├── video.zig # Window only
├── surface.zig # FlipMode only
└── overrides.zig # Manual defs (GPUShaderFormat)
```
## Technical Achievements
### Better Than Handwritten Code
1. **Type Safety**: Uses `*u32` instead of `[*c]u32` for output params
2. **Nullability**: Correct `?*` usage for nullable pointers
3. **Casting**: Minimal casts, only where needed
4. **Organization**: Methods grouped logically in opaque types
5. **Formatting**: Consistent, auto-formatted with AST
### Parser Architecture
```
Input (SDL_gpu.h)
Lexer/Parser → AST
Pattern Matching → Declarations
Type Conversion → Zig Types
Code Generation → Zig Source
AST Validation → Formatted Output
```
## Files Modified/Created
### Created
- `/lib/sdl3/parser/build.zig.zon` - Package definition
- `/lib/sdl3/parser/AGENTS.md` - Zig 0.15 guide
- `/lib/sdl3/parser/SUMMARY.md` - This file
- `/lib/sdl3/v2/gpu.zig` - Generated bindings
### Modified
- `/lib/sdl3/parser/src/codegen.zig` - Method grouping, ArrayList fixes
- `/lib/sdl3/parser/src/parser.zig` - AST rendering integration
- `/lib/sdl3/parser/src/types.zig` - Double pointer support
- `/lib/sdl3/build.zig` - Added regenerate-zig step
- `/lib/sdl3/build.zig.zon` - Added parser dependency
## Command Reference
```bash
# Build parser
cd lib/sdl3/parser
zig build
# Run tests
zig build test
# Generate GPU bindings
cd lib/sdl3
zig build regenerate-zig
# Manual generation
./parser/zig-out/bin/sdl-parser SDL/include/SDL3/SDL_gpu.h --output=v2/gpu.zig
```
## Comparison: Generated vs Handwritten
| Aspect | Generated (v2/gpu.zig) | Handwritten (src/gpu.zig) |
|--------|----------------------|--------------------------|
| Lines | 1,229 | 1,198 |
| Type Safety | ✅ Better | ⚠️ Uses [*c] |
| Nullability | ✅ Precise | ⚠️ Over-nullable |
| Methods | ✅ Grouped | ✅ Grouped |
| Casting | ✅ Minimal | ⚠️ Some unnecessary |
| Dependencies | ⚠️ Missing (planned) | ✅ Manual imports |
## Success Criteria Met
- ✅ Parses entire SDL_gpu.h without errors
- ✅ Generates syntactically valid Zig code
- ✅ All 169 declarations supported
- ✅ Better type safety than handwritten version
- ✅ Integrated into build system
- ✅ Tests passing
- ✅ Documentation complete
## Time Investment
- Parser development: ~4-5 hours
- Type system refinement: ~2 hours
- Method grouping: ~1 hour
- Zig 0.15 fixes: ~1 hour
- Documentation: ~1 hour
- **Total**: ~9-10 hours
## Impact
**Before**: Manual bindings, error-prone, difficult to maintain
**After**: Automated generation, type-safe, maintainable, better quality
**Line of Code Savings**:
- 1,229 lines auto-generated
- Can regenerate on SDL updates in seconds
- Can apply to other SDL headers (video, audio, etc.)
## Conclusion
The SDL3 parser successfully generates production-quality Zig bindings that are **safer and more ergonomic** than handwritten code. The only missing piece is dependency resolution, which has a clear implementation plan. The parser is ready for production use with manual dependency imports, and can be fully automated with the dependency resolution feature.
**Status**: 95% complete, production-ready with minor workarounds

File diff suppressed because it is too large Load Diff