Backlog/lib/sdl3/parser/docs/DEPENDENCY_FLOW.md

846 lines
23 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Dependency Resolution Flow - Technical Deep Dive
## Overview
This document traces the complete flow from parser entry point through dependency resolution to final output generation.
## Flow Diagram
```
main()
Parse Primary Header (SDL_gpu.h)
Analyze Dependencies
Extract Missing Types
Combine Declarations
Generate Output
```
## Detailed Step-by-Step Flow
### Phase 1: Parser Entry Point
**File**: `src/parser.zig::main()`
```zig
pub fn main() !void {
// 1. Setup
var gpa = std.heap.GeneralPurposeAllocator(.{}){};
const allocator = gpa.allocator();
// 2. Parse command line arguments
const header_path = args[1];
var output_file: ?[]const u8 = null;
var mock_output_file: ?[]const u8 = null;
// 3. Read the primary header file
const source = try std.fs.cwd().readFileAlloc(
allocator,
header_path,
10 * 1024 * 1024
);
defer allocator.free(source);
```
**Inputs**:
- Command line: `zig build run -- SDL_gpu.h --output=gpu.zig`
- Header file contents read into memory
**Outputs**:
- `source`: []const u8 - Full header file content
- `header_path`: []const u8 - Path for finding dependency headers
---
### Phase 2: Primary Header Parsing
**File**: `src/parser.zig::main()` continued
```zig
// 4. Parse declarations from primary header
var scanner = patterns.Scanner.init(allocator, source);
const decls = try scanner.scan();
// decls is now: []Declaration containing:
// - 13 opaque types (GPUDevice, GPUTexture, etc.)
// - 24 enums
// - 35 structs
// - 3 flags
// - 94 functions
```
**Process**:
1. `Scanner.init()` creates scanner with allocator and source
2. `scanner.scan()` iterates through source line by line
3. Tries each pattern: opaque, enum, struct, flags, function
4. Builds array of `Declaration` union variants
5. Each declaration owns its strings (allocated from scanner's allocator)
**Outputs**:
- `decls`: []Declaration - Array of 169 declarations from SDL_gpu.h
---
### Phase 3: Dependency Analysis Entry
**File**: `src/parser.zig::main()` continued
```zig
// 5. Create dependency resolver
var resolver = dependency_resolver.DependencyResolver.init(allocator);
defer resolver.deinit();
// 6. Analyze declarations to find missing types
try resolver.analyze(decls);
```
**What `DependencyResolver.init()` does**:
```zig
pub fn init(allocator: Allocator) DependencyResolver {
return .{
.allocator = allocator,
.referenced_types = std.StringHashMap(void).init(allocator),
.defined_types = std.StringHashMap(void).init(allocator),
};
}
```
Creates two HashMaps:
- `defined_types`: Types defined in primary header
- `referenced_types`: Types used in function signatures/struct fields
---
### Phase 4: Type Collection
**File**: `src/dependency_resolver.zig::DependencyResolver.analyze()`
```zig
pub fn analyze(self: *DependencyResolver, decls: []const Declaration) !void {
try self.collectDefinedTypes(decls); // Step 4a
try self.collectReferencedTypes(decls); // Step 4b
}
```
#### Step 4a: Collect Defined Types
```zig
fn collectDefinedTypes(self: *DependencyResolver, decls: []const Declaration) !void {
for (decls) |decl| {
const type_name = switch (decl) {
.opaque_type => |o| o.name, // e.g., "SDL_GPUDevice"
.enum_decl => |e| e.name, // e.g., "SDL_GPUPrimitiveType"
.struct_decl => |s| s.name, // e.g., "SDL_GPUViewport"
.flag_decl => |f| f.name, // e.g., "SDL_GPUTextureUsageFlags"
.function_decl => continue, // Functions don't define types
};
try self.defined_types.put(type_name, {});
}
}
```
**Result**: `defined_types` HashMap contains:
```
SDL_GPUDevice -> {}
SDL_GPUTexture -> {}
SDL_GPUViewport -> {}
SDL_GPUPrimitiveType -> {}
... (166 more entries)
```
#### Step 4b: Collect Referenced Types
```zig
fn collectReferencedTypes(self: *DependencyResolver, decls: []const Declaration) !void {
for (decls) |decl| {
switch (decl) {
.function_decl => |func| {
// Scan return type
try self.scanType(func.return_type);
// Scan each parameter type
for (func.params) |param| {
try self.scanType(param.type_name);
}
},
.struct_decl => |struct_decl| {
// Scan each field type
for (struct_decl.fields) |field| {
try self.scanType(field.type_name);
}
},
else => {},
}
}
}
```
**Example**: Function signature processing
```c
// C function:
bool SDL_WindowSupportsGPUSwapchain(SDL_GPUDevice *device, SDL_Window *window)
// Parser sees:
.function_decl = {
.return_type = "bool",
.params = [
{ .type_name = "SDL_GPUDevice *" },
{ .type_name = "SDL_Window *" }
]
}
```
**For each type string, calls `scanType()`**:
---
### Phase 5: Type Extraction & Normalization
**File**: `src/dependency_resolver.zig::scanType()`
```zig
fn scanType(self: *DependencyResolver, type_str: []const u8) !void {
// Extract base type from decorated string
const base_type = extractBaseType(type_str);
if (base_type.len > 0 and isSDLType(base_type)) {
// Only add if not already present (deduplicate)
if (!self.referenced_types.contains(base_type)) {
// Must own the string (type_str may be freed)
const owned = try self.allocator.dupe(u8, base_type);
try self.referenced_types.put(owned, {});
}
}
}
```
#### Example: Type Extraction Process
**Input**: `"SDL_Window *"`
**Step-by-step through `extractBaseType()`**:
```zig
fn extractBaseType(type_str: []const u8) []const u8 {
var result = "SDL_Window *";
// Loop 1: Remove leading qualifiers
result = std.mem.trim(u8, result, " \t"); // "SDL_Window *"
// No leading "const", "?", "*", etc.
// Loop 2: Remove trailing qualifiers
result = std.mem.trim(u8, result, " \t"); // "SDL_Window *"
// Check trailing "*"
if (std.mem.endsWith(u8, result, "*")) {
result = result[0..result.len-1]; // "SDL_Window "
continue;
}
result = std.mem.trim(u8, result, " \t"); // "SDL_Window"
return "SDL_Window";
}
```
**Output**: `"SDL_Window"` (clean type name)
**More Examples**:
```
"?*SDL_GPUDevice" -> "SDL_GPUDevice"
"*const SDL_Rect" -> "SDL_Rect"
"SDL_GPUBuffer *const *" -> "SDL_GPUBuffer"
"[*c]const u8" -> "u8"
"SDL_FColor" -> "SDL_FColor"
```
#### SDL Type Detection
```zig
fn isSDLType(type_str: []const u8) bool {
// Check for SDL_ prefix
if (std.mem.startsWith(u8, type_str, "SDL_")) {
return true;
}
// Check known Zig-ified names
const known_types = [_][]const u8{
"Window", "Rect", "FColor", "FlipMode",
"PropertiesID", "Surface", ...
};
for (known_types) |known| {
if (std.mem.eql(u8, type_str, known)) {
return true;
}
}
return false; // Primitive type like "bool", "u32"
}
```
**Result**: `referenced_types` HashMap contains:
```
SDL_Window -> {}
SDL_Rect -> {}
SDL_FColor -> {}
SDL_FlipMode -> {}
SDL_PropertiesID -> {}
SDL_GPUShaderFormat -> {}
```
---
### Phase 6: Missing Type Calculation
**File**: `src/parser.zig::main()` continued
```zig
// 7. Get missing types (referenced but not defined)
const missing_types = try resolver.getMissingTypes(allocator);
defer {
for (missing_types) |t| allocator.free(t);
allocator.free(missing_types);
}
```
**File**: `src/dependency_resolver.zig::getMissingTypes()`
```zig
pub fn getMissingTypes(self: *DependencyResolver, allocator: Allocator) ![][]const u8 {
var missing = std.ArrayList([]const u8){};
var it = self.referenced_types.keyIterator();
while (it.next()) |key| {
// Check if type is NOT in defined_types
if (!self.defined_types.contains(key.*)) {
// This is a missing type - need to find it
try missing.append(allocator, try allocator.dupe(u8, key.*));
}
}
return try missing.toOwnedSlice(allocator);
}
```
**Logic**:
```
referenced_types = {SDL_Window, SDL_Rect, SDL_FColor, ...}
defined_types = {SDL_GPUDevice, SDL_GPUTexture, ...}
missing_types = referenced_types - defined_types
= {SDL_Window, SDL_Rect, SDL_FColor, SDL_FlipMode,
SDL_PropertiesID, SDL_GPUShaderFormat}
```
**Output**: Array of 6 strings (owned by caller)
---
### Phase 7: Include Header Parsing
**File**: `src/parser.zig::main()` continued
```zig
if (missing_types.len > 0) {
// 8. Parse #include directives from source
const includes = try dependency_resolver.parseIncludes(allocator, source);
defer {
for (includes) |inc| allocator.free(inc);
allocator.free(includes);
}
```
**File**: `src/dependency_resolver.zig::parseIncludes()`
```zig
pub fn parseIncludes(allocator: Allocator, source: []const u8) ![][]const u8 {
var includes = std.ArrayList([]const u8){};
var lines = std.mem.splitScalar(u8, source, '\n');
while (lines.next()) |line| {
const trimmed = std.mem.trim(u8, line, " \t\r");
// Match: #include <SDL3/SDL_something.h>
if (std.mem.startsWith(u8, trimmed, "#include <SDL3/")) {
const after_open = "#include <SDL3/".len;
if (std.mem.indexOf(u8, trimmed[after_open..], ">")) |end| {
const header_name = trimmed[after_open..][0..end];
try includes.append(allocator, try allocator.dupe(u8, header_name));
}
}
}
return try includes.toOwnedSlice(allocator);
}
```
**Example**: From SDL_gpu.h header:
```c
#include <SDL3/SDL_stdinc.h>
#include <SDL3/SDL_pixels.h>
#include <SDL3/SDL_properties.h>
#include <SDL3/SDL_rect.h>
#include <SDL3/SDL_surface.h>
#include <SDL3/SDL_video.h>
```
**Output**: Array of strings:
```
["SDL_stdinc.h", "SDL_pixels.h", "SDL_properties.h",
"SDL_rect.h", "SDL_surface.h", "SDL_video.h"]
```
---
### Phase 8: Dependency Type Extraction
**File**: `src/parser.zig::main()` continued
```zig
// 9. Determine header directory
const header_dir = std.fs.path.dirname(header_path) orelse ".";
// e.g., "../SDL/include/SDL3"
var dependency_decls = std.ArrayList(patterns.Declaration){};
defer {
for (dependency_decls.items) |dep_decl| {
freeDeclDeep(allocator, dep_decl);
}
dependency_decls.deinit(allocator);
}
// 10. For each missing type, search dependency headers
for (missing_types) |missing_type| {
var found = false;
// Try each included header
for (includes) |include| {
// 10a. Build full path
const dep_path = try std.fs.path.join(
allocator,
&[_][]const u8{ header_dir, include }
);
defer allocator.free(dep_path);
// e.g., "../SDL/include/SDL3/SDL_pixels.h"
// 10b. Read dependency header
const dep_source = std.fs.cwd().readFileAlloc(
allocator,
dep_path,
10 * 1024 * 1024
) catch continue; // Skip if can't read
defer allocator.free(dep_source);
// 10c. Extract type from this header
if (try dependency_resolver.extractTypeFromHeader(
allocator,
dep_source,
missing_type
)) |dep_decl| {
try dependency_decls.append(allocator, dep_decl);
std.debug.print(" ✓ Found {s} in {s}\n",
.{missing_type, include});
found = true;
break; // Found it, stop searching
}
}
if (!found) {
std.debug.print(" ⚠ Warning: Could not find {s}\n",
.{missing_type});
}
}
```
**Search Algorithm**:
```
For missing_type "SDL_Window":
Try SDL_stdinc.h -> Not found
Try SDL_pixels.h -> Not found
Try SDL_properties.h -> Not found
Try SDL_rect.h -> Not found
Try SDL_surface.h -> Not found
Try SDL_video.h -> FOUND! ✓
```
---
### Phase 9: Type Extraction from Header
**File**: `src/dependency_resolver.zig::extractTypeFromHeader()`
```zig
pub fn extractTypeFromHeader(
allocator: Allocator,
header_source: []const u8,
type_name: []const u8, // e.g., "SDL_Window"
) !?Declaration {
// 1. Parse the entire dependency header
var scanner = patterns.Scanner.init(allocator, header_source);
const all_decls = try scanner.scan();
defer {
for (all_decls) |decl| {
freeDeclaration(allocator, decl);
}
allocator.free(all_decls);
}
// 2. Search for matching type
for (all_decls) |decl| {
const decl_name = switch (decl) {
.opaque_type => |o| o.name,
.enum_decl => |e| e.name,
.struct_decl => |s| s.name,
.flag_decl => |f| f.name,
else => continue,
};
// 3. Found it!
if (std.mem.eql(u8, decl_name, type_name)) {
// 4. Deep clone so caller owns it
return try cloneDeclaration(allocator, decl);
}
}
return null; // Not found in this header
}
```
**Example**: Searching SDL_video.h for SDL_Window
1. Parse SDL_video.h → 50+ declarations
2. Iterate through all declarations
3. Find: `.opaque_type = { .name = "SDL_Window", ... }`
4. Clone the declaration (deep copy all strings)
5. Return the clone
6. Free all the temporary declarations from parsing
**Cloning Process**:
```zig
fn cloneDeclaration(allocator: Allocator, decl: Declaration) !Declaration {
return switch (decl) {
.opaque_type => |o| .{
.opaque_type = .{
.name = try allocator.dupe(u8, o.name), // Own the string
.doc_comment = if (o.doc_comment) |doc|
try allocator.dupe(u8, doc) else null,
},
},
// ... similar for enum, struct, flags
};
}
```
**Why clone?** The parsed declarations from `scanner.scan()` are freed after this function returns. We need owned copies that live until code generation.
---
### Phase 10: Declaration Combining
**File**: `src/parser.zig::main()` continued
```zig
// 11. Combine dependency declarations with primary
var all_decls = std.ArrayList(patterns.Declaration){};
defer all_decls.deinit(allocator);
// IMPORTANT: Dependencies FIRST!
try all_decls.appendSlice(allocator, dependency_decls.items);
try all_decls.appendSlice(allocator, decls);
```
**Result**: Combined array
```
all_decls = [
// Dependencies (4 items)
{ .struct_decl = SDL_FColor },
{ .enum_decl = SDL_FlipMode },
{ .struct_decl = SDL_Rect },
{ .opaque_type = SDL_Window },
// Primary header (169 items)
{ .opaque_type = SDL_GPUDevice },
{ .enum_decl = SDL_GPUPrimitiveType },
... (167 more)
]
```
**Why dependencies first?** Types must be defined before they're used. Since primary header references dependency types, dependencies must come first.
---
### Phase 11: Code Generation
**File**: `src/parser.zig::main()` continued
```zig
// 12. Generate Zig code from all declarations
const output = try codegen.CodeGen.generate(allocator, all_decls.items);
defer allocator.free(output);
```
**File**: `src/codegen.zig::CodeGen.generate()` (simplified)
```zig
pub fn generate(allocator: Allocator, decls: []const Declaration) ![]const u8 {
var buf = std.ArrayList(u8){};
// Header
try buf.appendSlice(allocator, "pub const c = @import(\"c.zig\").c;\n\n");
// Generate each declaration
for (decls) |decl| {
switch (decl) {
.opaque_type => |o| {
try buf.appendSlice(allocator, "pub const ");
try buf.appendSlice(allocator, stripSDLPrefix(o.name));
try buf.appendSlice(allocator, " = opaque {};\n");
},
.struct_decl => |s| {
try generateStruct(allocator, &buf, s);
},
// ... other types
}
}
return try buf.toOwnedSlice(allocator);
}
```
**Output** (excerpt):
```zig
pub const c = @import("c.zig").c;
pub const FColor = extern struct {
r: f32,
g: f32,
b: f32,
a: f32,
};
pub const Window = opaque {};
pub const GPUDevice = opaque {
pub inline fn windowSupportsGPU(
gpudevice: *GPUDevice,
window: ?*Window, // ✓ Window is defined above!
) bool {
return c.SDL_WindowSupportsGPUDevice(gpudevice, window);
}
};
```
---
### Phase 12: AST Validation & Formatting
**File**: `src/parser.zig::main()` continued
```zig
// 13. Parse generated code as Zig AST
const output_z = try allocator.dupeZ(u8, output);
defer allocator.free(output_z);
var ast = try std.zig.Ast.parse(allocator, output_z, .zig);
defer ast.deinit(allocator);
// 14. Check for syntax errors
if (ast.errors.len > 0) {
std.debug.print("\nError: {d} syntax errors\n", .{ast.errors.len});
for (ast.errors) |err| {
const loc = ast.tokenLocation(0, err.token);
std.debug.print(" Line {d}: {s}\n",
.{ loc.line + 1, @tagName(err.tag) });
}
return error.InvalidSyntax;
}
// 15. Format using Zig's formatter
const formatted_output = try ast.renderAlloc(allocator);
defer allocator.free(formatted_output);
```
**Why validate?** Catch codegen bugs early. If generated code doesn't parse, we know immediately.
**Why format?** Zig's formatter ensures consistent style, proper indentation, and canonical formatting.
---
### Phase 13: Output Writing
**File**: `src/parser.zig::main()` continued
```zig
// 16. Write to file or stdout
if (output_file) |file_path| {
try std.fs.cwd().writeFile(.{
.sub_path = file_path,
.data = formatted_output,
});
std.debug.print("Generated: {s}\n", .{file_path});
} else {
_ = try std.posix.write(std.posix.STDOUT_FILENO, formatted_output);
}
```
---
## Memory Management Flow
### Allocations
1. **Primary header source**: Freed at end of main()
2. **Primary declarations**: Freed at end of main() (with deep free)
3. **Dependency resolver HashMaps**: Freed in resolver.deinit()
4. **HashMap keys** (in referenced_types): Freed in resolver.deinit()
5. **Missing types array**: Freed explicitly after use
6. **Includes array**: Freed explicitly after use
7. **Dependency header sources**: Freed immediately after extraction
8. **Temporary parsed declarations**: Freed immediately in extractTypeFromHeader()
9. **Cloned dependency declarations**: Freed at end of scope
10. **Generated output**: Freed after writing
11. **Formatted output**: Freed after writing
### Ownership Rules
- **Scanner owns strings** during parsing (from its allocator)
- **Cloned declarations own strings** after extraction (allocated explicitly)
- **HashMap owns keys** in referenced_types (duped when inserted)
- **Caller owns result** of getMissingTypes(), parseIncludes()
---
## Error Handling Flow
### Errors That Fail
```zig
// Fatal errors - exit immediately
- File not found (primary header)
- Out of memory
- Invalid syntax in generated code (optional)
```
### Errors That Warn
```zig
// Warnings - continue execution
- Dependency header not readable continue with next header
- Type not found in any header print warning, continue
- Struct parsing errors generate partial output
```
### Example Error Flow
```
Parse SDL_gpu.h
Missing type: SDL_Window
Try SDL_pixels.h → catch FileNotFound → continue
Try SDL_video.h → Success! → break
Missing type: SDL_Unknown
Try all headers → Not found → print warning
Continue with partial results
```
---
## Performance Characteristics
### Time Complexity
- Primary parsing: O(n) where n = source lines
- Type collection: O(d) where d = declarations
- Missing type detection: O(r) where r = referenced types
- Type extraction: O(h × d) where h = headers, d = declarations per header
- Overall: O(n + d + r + h×d) ≈ O(n) for typical cases
### Space Complexity
- Primary declarations: O(d)
- Dependency declarations: O(m) where m = missing types
- HashMaps: O(t) where t = total unique types
- Peak memory: ~2-5MB for SDL_gpu.h
### Optimization Points
1. **Cache parsed headers** - Currently re-parse for each missing type
2. **Early exit** - Stop searching after finding type
3. **String interning** - Deduplicate type name strings
4. **Lazy loading** - Only parse dependencies if missing types detected
---
## Testing the Flow
### Unit Test Example
```zig
test "complete dependency flow" {
const source =
\\typedef struct SDL_Type SDL_Type;
\\extern void SDL_Func(SDL_External *param);
;
// Phase 1: Parse
var scanner = Scanner.init(allocator, source);
const decls = try scanner.scan();
// Phase 2: Analyze
var resolver = DependencyResolver.init(allocator);
defer resolver.deinit();
try resolver.analyze(decls);
// Phase 3: Get missing
const missing = try resolver.getMissingTypes(allocator);
defer allocator.free(missing);
// Verify: SDL_External is missing
try testing.expectEqual(@as(usize, 1), missing.len);
try testing.expectEqualStrings("SDL_External", missing[0]);
}
```
### Integration Test
```bash
# Create test header with dependency
echo 'typedef struct Dep Dep;' > dep.h
echo '#include "dep.h"' > main.h
echo 'void func(Dep *d);' >> main.h
# Parse with dependency resolution
zig build run -- main.h --output=out.zig
# Verify output contains Dep
grep 'pub const Dep' out.zig
```
---
## Summary
The dependency resolution flow is:
1. **Parse** primary header → get declarations
2. **Analyze** declarations → find referenced vs defined types
3. **Calculate** missing = referenced - defined
4. **Extract** #include directives from source
5. **Search** dependency headers for missing types
6. **Clone** found declarations (deep copy)
7. **Combine** dependency + primary declarations
8. **Generate** Zig code with all types
9. **Validate** and format using Zig AST
10. **Output** to file or stdout
Each phase has clear inputs/outputs, proper memory management, and graceful error handling.