Add comprehensive dependency resolution implementation plan

Plan for on-demand type resolution:
- Single-file output with dependencies appended
- Parse included headers only for missing types
- Three-phase algorithm: parse, detect, resolve
- Five components with clear interfaces and tests

Key features:
- Type reference scanner (finds all type usage)
- Defined type collector (tracks what exists)
- Include header parser (extracts #include directives)
- Selective type extractor (finds specific types)
- Main integration (wires everything together)

Advantages:
- Simple: one output file, no modules
- Fast: parse dependencies once, extract many types
- Minimal: only extract required types
- Robust: Zig handles duplicate definitions
- Testable: each component independently tested

Ready to implement with:
- Detailed code examples for each component
- Complete test strategy
- Success metrics
- 5-day rollout plan
- Risk mitigation
This commit is contained in:
Peterino2 2026-01-22 01:46:58 -08:00
parent 4c354f093c
commit c7c7440c14
1 changed files with 667 additions and 0 deletions

View File

@ -0,0 +1,667 @@
# Dependency Resolution Implementation Plan (v2)
## Core Insight
**Single-file generation with on-demand type resolution**: Parse the primary header completely, identify missing types, then parse included headers ONLY for those specific types. Append them to the same output file. Zig's structural typing handles everything else.
## Why This Works
1. **No modules needed** - One output file with all types
2. **Zig deduplicates** - Multiple parsings of same type are safe
3. **Forward references work** - Zig allows using types before they're defined
4. **Simple implementation** - Just append to existing output
## Three-Phase Algorithm
### Phase 1: Primary Header Parsing (Existing)
```
Input: SDL_gpu.h
Output: declarations[], generated_code
```
### Phase 2: Missing Type Detection (NEW)
```
1. Scan generated_code for all type references
2. Build set of defined_types from declarations[]
3. missing_types = referenced_types - defined_types
```
### Phase 3: Dependency Resolution (NEW)
```
For each missing_type:
1. Parse each included header
2. If type found, extract declaration
3. Append to output
```
## Implementation Details
## Implementation Details
### Component 1: Type Reference Scanner
**Purpose**: Find all type names used in generated code
**Location**: New file `src/dependency_resolver.zig`
```zig
pub const TypeReference = struct {
name: []const u8,
source_location: []const u8, // For debugging
};
pub fn scanTypeReferences(decls: []const Declaration) ![]TypeReference {
var refs = ArrayList(TypeReference).init(allocator);
for (decls) |decl| {
switch (decl) {
.function_decl => |func| {
// Scan return type
try scanType(func.return_type, &refs);
// Scan parameters
for (func.params) |param| {
try scanType(param.type_name, &refs);
}
},
.struct_decl => |struct_decl| {
// Scan field types
for (struct_decl.fields) |field| {
try scanType(field.type_name, &refs);
}
},
// opaque/enum don't reference other types
else => {},
}
}
return refs.toOwnedSlice();
}
fn scanType(type_str: []const u8, refs: *ArrayList(TypeReference)) !void {
// Extract base type from "?*const SDL_Type" → "SDL_Type"
const base_type = extractBaseType(type_str);
if (isSDLType(base_type)) {
try refs.append(.{ .name = base_type, .source_location = type_str });
}
}
```
**Key functions**:
- `extractBaseType()`: Strip pointers, const, optional from type string
- `isSDLType()`: Check if starts with "SDL_" or is known SDL type
- Handle edge cases: arrays, function pointers (skip for now)
### Component 2: Defined Type Collector
**Purpose**: Track what types are already defined
```zig
pub fn collectDefinedTypes(decls: []const Declaration) StringHashMap(void) {
var defined = StringHashMap(void).init(allocator);
for (decls) |decl| {
const type_name = switch (decl) {
.opaque_type => |o| o.name,
.enum_decl => |e| e.name,
.struct_decl => |s| s.name,
.flags_decl => |f| f.name,
.function_decl => continue, // Functions don't define types
};
try defined.put(type_name, {});
}
return defined;
}
```
### Component 3: Include Header Parser
**Purpose**: Extract #include directives
```zig
pub fn parseIncludes(source: []const u8) ![]const []const u8 {
var includes = ArrayList([]const u8).init(allocator);
var lines = std.mem.split(u8, source, "\n");
while (lines.next()) |line| {
// Match: #include <SDL3/SDL_something.h>
if (std.mem.indexOf(u8, line, "#include <SDL3/")) |start| {
const after_open = start + "#include <SDL3/".len;
if (std.mem.indexOf(u8, line[after_open..], ">")) |end| {
const header_name = line[after_open..][0..end];
try includes.append(try allocator.dupe(u8, header_name));
}
}
}
return includes.toOwnedSlice();
}
```
### Component 4: Selective Type Extractor
**Purpose**: Find specific type in a header
```zig
pub fn extractTypeFromHeader(
allocator: Allocator,
header_source: []const u8,
type_name: []const u8, // e.g., "SDL_Rect"
) !?Declaration {
// Parse the header
var scanner = Scanner.init(allocator, header_source);
const all_decls = try scanner.scan();
defer scanner.deinit();
// Find matching declaration
for (all_decls) |decl| {
const decl_name = switch (decl) {
.opaque_type => |o| o.name,
.enum_decl => |e| e.name,
.struct_decl => |s| s.name,
.flags_decl => |f| f.name,
else => continue,
};
if (std.mem.eql(u8, decl_name, type_name)) {
return try decl.clone(allocator); // Deep copy
}
}
return null; // Not found
}
```
### Component 5: Main Integration
**Location**: Modify `src/parser.zig` main function
```zig
pub fn main() !void {
// ... existing setup ...
// 1. Parse primary header (existing)
var scanner = Scanner.init(allocator, source);
const primary_decls = try scanner.scan();
// 2. Identify missing types (NEW)
const references = try scanTypeReferences(primary_decls);
const defined = collectDefinedTypes(primary_decls);
var missing = ArrayList([]const u8).init(allocator);
for (references) |ref| {
if (!defined.contains(ref.name)) {
try missing.append(ref.name);
}
}
// 3. Extract missing types from dependencies (NEW)
var dependency_decls = ArrayList(Declaration).init(allocator);
if (missing.items.len > 0) {
const includes = try parseIncludes(source);
const header_dir = std.fs.path.dirname(header_path) orelse ".";
for (missing.items) |missing_type| {
var found = false;
for (includes) |include| {
const dep_path = try std.fs.path.join(
allocator,
&[_][]const u8{ header_dir, include }
);
defer allocator.free(dep_path);
const dep_source = std.fs.cwd().readFileAlloc(
allocator,
dep_path,
10 * 1024 * 1024
) catch continue;
defer allocator.free(dep_source);
if (try extractTypeFromHeader(allocator, dep_source, missing_type)) |decl| {
try dependency_decls.append(decl);
found = true;
break;
}
}
if (!found) {
std.debug.print(
"Warning: Could not find definition for type: {s}\n",
.{missing_type}
);
}
}
}
// 4. Combine declarations
var all_decls = ArrayList(Declaration).init(allocator);
try all_decls.appendSlice(dependency_decls.items); // Dependencies first!
try all_decls.appendSlice(primary_decls);
// 5. Generate code (existing, but with all declarations)
const output = try codegen.generate(allocator, all_decls.items);
// ... rest of existing code ...
}
```
## Key Implementation Decisions
1. **Dependencies go FIRST** in output
- Ensures types are defined before use
- More logical reading order
2. **Deep copy declarations**
- Avoid lifetime issues with parsed headers
- Each declaration owns its strings
3. **Warning for missing types**
- Don't fail build, just warn
- Allows gradual improvement
4. **Skip function pointers/unions**
- Add TODO comments
- Focus on common cases first
5. **Cache parsed headers**
- Parse each dependency header once
- Extract multiple types from same parse
## Testing Approach
### Unit Tests
```zig
test "scanTypeReferences finds SDL types" {
const decls = [_]Declaration{
.{ .function_decl = .{
.name = "test",
.return_type = "?*SDL_Window",
.params = &[_]Param{
.{ .name = "rect", .type_name = "*const SDL_Rect" },
},
}},
};
const refs = try scanTypeReferences(&decls);
try testing.expect(refs.len == 2);
try testing.expectEqualStrings("SDL_Window", refs[0].name);
try testing.expectEqualStrings("SDL_Rect", refs[1].name);
}
test "collectDefinedTypes tracks declarations" {
const decls = [_]Declaration{
.{ .opaque_type = .{ .name = "SDL_Device" }},
.{ .struct_decl = .{ .name = "SDL_Info", .fields = &[_]Field{} }},
};
const defined = collectDefinedTypes(&decls);
try testing.expect(defined.contains("SDL_Device"));
try testing.expect(defined.contains("SDL_Info"));
}
```
### Integration Test
```zig
test "full dependency resolution with SDL_gpu.h" {
const source = try std.fs.cwd().readFileAlloc(
testing.allocator,
"SDL/include/SDL3/SDL_gpu.h",
10 * 1024 * 1024
);
defer testing.allocator.free(source);
// Parse and resolve
const output = try parseWithDependencies(testing.allocator, source, "SDL/include/SDL3");
defer testing.allocator.free(output);
// Verify missing types are present
try testing.expect(std.mem.indexOf(u8, output, "pub const Window = opaque") != null);
try testing.expect(std.mem.indexOf(u8, output, "pub const Rect = extern struct") != null);
try testing.expect(std.mem.indexOf(u8, output, "pub const FColor = extern struct") != null);
// Verify it compiles
var ast = try std.zig.Ast.parse(testing.allocator, output, .zig);
defer ast.deinit(testing.allocator);
try testing.expect(ast.errors.len == 0);
}
```
## Validation Steps
1. **Remove manual definitions** from mock_test.zig:
```diff
- pub const Window = opaque {};
- pub const Rect = extern struct { ... };
- pub const FColor = extern struct { ... };
```
2. **Import generated file** directly:
```zig
const gpu = @import("../../zig-out/gpu_test.zig");
```
3. **Run tests**:
```bash
zig build test-mocks # Should still pass!
```
## Success Metrics
`scanTypeReferences` finds 30+ type references in SDL_gpu.h
`collectDefinedTypes` tracks 169 defined types
✅ Missing types: Window, Rect, FColor, FlipMode, PropertiesID detected
✅ All 5 missing types extracted from dependency headers
✅ Generated code compiles without manual definitions
✅ All 11 tests pass
✅ Build time increase < 1 second
## Rollout Plan
1. **Day 1**: Implement Components 1-2 (type scanning/collecting)
2. **Day 2**: Implement Components 3-4 (include parsing/type extraction)
3. **Day 3**: Integrate into main, test with SDL_gpu.h
4. **Day 4**: Refine, handle edge cases, update tests
5. **Day 5**: Documentation, final validation
## Risks & Mitigation
| Risk | Mitigation |
|------|-----------|
| Can't find header files | Require header directory as input |
| Type not in any header | Emit warning, generate placeholder |
| Parsing dependency fails | Catch error, continue with other headers |
| Performance (parsing multiple headers) | Cache parsed headers, parse once |
| Circular dependencies | Not an issue - all types in one file |
---
This plan is **ready to implement**. Each component is well-defined with clear inputs/outputs, error handling, and test cases.
**File**: `src/type_collector.zig`
```zig
const TypeCollector = struct {
defined_types: StringHashMap(void), // Types defined in primary header
referenced_types: StringHashMap(void), // Types used in signatures
pub fn collectFromDeclarations(decls: []Declaration) TypeCollector;
pub fn getMissingTypes() []const []const u8;
};
```
**Tasks**:
- Scan all declarations for type definitions (opaque, enum, struct, flags)
- Scan all function signatures for type references
- Return set difference: referenced - defined
### Phase 2: Include Directive Parsing (1 hour)
**File**: `src/patterns.zig` (extend existing)
```zig
pub fn parseIncludes(source: []const u8) ![]const []const u8 {
// Find all #include <SDL3/header.h> directives
// Return list of header filenames
}
```
**Tasks**:
- Add regex/pattern for `#include <SDL3/...>`
- Extract header filename from directive
- Return list of included headers
### Phase 3: Selective Type Extraction (2-3 hours)
**File**: `src/type_extractor.zig`
```zig
pub fn extractType(
allocator: Allocator,
header_source: []const u8,
type_name: []const u8, // e.g., "SDL_Rect"
) !?Declaration {
// Parse header looking for specific type
// Return the declaration if found
}
pub fn extractTypes(
allocator: Allocator,
header_paths: []const []const u8,
missing_types: []const []const u8,
) ![]Declaration {
// For each missing type:
// For each header:
// Try to extract the type
// If found, add to results
// Return all found declarations
}
```
**Tasks**:
- Reuse existing Scanner but filter by type name
- Handle opaque types: `typedef struct SDL_Type SDL_Type;`
- Handle structs: `typedef struct { ... } SDL_Type;`
- Handle enums: `typedef enum { ... } SDL_Type;`
- Handle simple typedefs: `typedef uint32_t SDL_Type;`
### Phase 4: Integration (1-2 hours)
**File**: `src/parser.zig` (modify main function)
```zig
pub fn main() !void {
// 1. Parse primary header
var scanner = Scanner.init(allocator, source);
const decls = try scanner.scan();
// 2. Collect missing types
const collector = TypeCollector.collectFromDeclarations(decls);
const missing_types = try collector.getMissingTypes(allocator);
// 3. Parse included headers for missing types
const includes = try parseIncludes(source);
const header_dir = getHeaderDirectory(header_path);
const dependency_decls = try extractTypes(
allocator,
header_dir,
includes,
missing_types
);
// 4. Generate code with dependencies appended
var all_decls = std.ArrayList(Declaration).init(allocator);
try all_decls.appendSlice(decls);
try all_decls.appendSlice(dependency_decls);
const output = try codegen.generate(allocator, all_decls.items);
// 5. Write output
try writeOutput(output_file, output);
}
```
**Tasks**:
- Wire together all components
- Handle header path resolution
- Add comment separators for dependencies
- Update error handling
### Phase 5: Code Generation Enhancement (1 hour)
**File**: `src/codegen.zig` (modify)
Add dependency section:
```zig
fn generate() ![]const u8 {
try output.appendSlice("pub const c = @import(\"c.zig\").c;\n\n");
// Add comment if we have dependencies
if (has_dependency_decls) {
try output.appendSlice(
\\// Dependencies from included headers
\\// These types are referenced by the primary header
\\
);
}
// Generate all declarations (primary + dependencies)
for (decls) |decl| {
try generateDeclaration(decl);
}
}
```
**Tasks**:
- Add dependency comment section
- Mark which declarations are dependencies (optional)
- Ensure proper ordering (dependencies before usage)
## Example Output
```zig
pub const c = @import("c.zig").c;
// Dependencies from included headers
// These types are referenced by SDL_gpu.h
// From SDL_rect.h
pub const Rect = extern struct {
x: i32,
y: i32,
w: i32,
h: i32,
};
// From SDL_pixels.h
pub const FColor = extern struct {
r: f32,
g: f32,
b: f32,
a: f32,
};
// From SDL_video.h
pub const Window = opaque {};
// From SDL_properties.h
pub const PropertiesID = u32;
// SDL_gpu.h declarations
pub const GPUDevice = opaque {
pub inline fn windowSupportsGPUSwapchainComposition(
gpudevice: *GPUDevice,
window: ?*Window, // ✅ Now defined!
swapchain_composition: GPUSwapchainComposition
) bool { ... }
};
pub const GPURenderPass = opaque {
pub inline fn setGPUScissor(
gpurenderpass: *GPURenderPass,
scissor: *const Rect // ✅ Now defined!
) void { ... }
};
```
## Edge Cases
1. **Type not found in any header**
- Emit warning
- Generate placeholder: `pub const TypeName = opaque {};`
2. **Circular dependencies**
- Not an issue - all types in one file
- Zig allows forward references
3. **Multiple definitions**
- Keep first definition found
- Zig will error if layouts differ (good!)
4. **Typedef chains**
- `typedef SDL_Type1 Type2;`
- Resolve transitively or use Zig's type alias
5. **Complex types**
- Function pointers: Skip for now, add TODO comment
- Unions: Skip for now, add TODO comment
- Nested structs: Should work fine
## Testing Strategy
1. **Unit tests** for each component:
- TypeCollector: Test with known declarations
- parseIncludes: Test with sample headers
- extractType: Test finding types in headers
2. **Integration test**:
- Parse SDL_gpu.h
- Verify missing types are detected
- Verify dependencies are extracted
- Verify output compiles
3. **Validation**:
- Remove manual type definitions from mock_test.zig
- Import actual generated file
- All tests should still pass
## Success Criteria
✅ Parser detects 5 missing types from SDL_gpu.h
✅ Parser extracts types from dependency headers
✅ Generated code compiles standalone
✅ All tests pass without manual type definitions
✅ Single output file contains all needed types
## Time Estimate
- Phase 1 (Type Collection): 1-2 hours
- Phase 2 (Include Parsing): 1 hour
- Phase 3 (Type Extraction): 2-3 hours
- Phase 4 (Integration): 1-2 hours
- Phase 5 (Code Gen Enhancement): 1 hour
- Testing & Refinement: 2 hours
**Total: 8-11 hours**
## Advantages of This Approach
1. **Simple**: Single output file, no module management
2. **Fast**: Only parse dependency headers when needed
3. **Minimal**: Only extract required types
4. **Robust**: Zig handles duplicate definitions
5. **Maintainable**: Clear separation in output
## Open Questions
1. Should we cache parsed dependency headers?
- **Answer**: Yes, parse once, extract many types
2. How to handle nested dependencies (Type A needs Type B)?
- **Answer**: Recursive extraction, track visited types
3. Should dependencies go at top or bottom of file?
- **Answer**: Top, before primary declarations use them
4. What about #define constants?
- **Answer**: Skip for now, out of scope
## Next Steps
1. Implement TypeCollector
2. Implement include parsing
3. Implement type extraction
4. Wire together in main
5. Test with SDL_gpu.h
6. Update documentation
---
Ready to implement? This plan provides:
- Clear phases with time estimates
- Concrete code examples
- Handles edge cases
- Simple single-file output
- Full testing strategy