adding claude helper files and documentation, also testing parallel-job
This commit is contained in:
parent
e119dac3e1
commit
efc11fcba9
|
|
@ -0,0 +1,128 @@
|
|||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
Backlog is a game engine written in Zig (version 0.14) with a modular architecture. The engine supports cross-platform development with Windows, macOS, and Linux targets, and includes support for both static and dynamic module loading.
|
||||
|
||||
## Build System
|
||||
|
||||
The project uses Zig's build system with a custom `BuildSystem` wrapper:
|
||||
|
||||
### Key Build Commands
|
||||
|
||||
- `zig build` - Build the engine and default projects
|
||||
- `zig build -Dcookshaders` - Build with shader compilation enabled
|
||||
- `zig build -Dstatic_build` - Force static linking (required for Linux)
|
||||
- `zig build tools` - Install development tools (gltf2ozz, spirv-reflect)
|
||||
- `zig build gltf2ozz -- [args]` - Run GLTF animation converter
|
||||
- `zig build spv-reflect -- [args]` - Run SPIRV reflection tool
|
||||
- `python tools/scripts/cookShaders.py` - Compile shaders from HLSL to SPIR-V, MSL, and DXIL formats, must be run after editing shader files
|
||||
|
||||
### Project-Specific Commands
|
||||
|
||||
From the `projects/` directory:
|
||||
- `zig build` - Build sample game and tools
|
||||
- `zig build run-sampleGame` - Run the sample game
|
||||
- `zig build run-newProject` - Run the project creation tool
|
||||
|
||||
### Testing
|
||||
|
||||
- `zig build test` - Run all tests (individual modules have their own test files in `tests/` subdirectories)
|
||||
|
||||
### Setup Commands
|
||||
|
||||
- `python tools/scripts/first-time-setup.py` - Initial setup script
|
||||
|
||||
## Architecture
|
||||
|
||||
### Engine Modules
|
||||
|
||||
The engine is organized into these core modules:
|
||||
|
||||
- **core** - Foundation systems (ECS, logging, memory tracking, jobs, scripting)
|
||||
- **platform** - Cross-platform windowing and input handling
|
||||
- **assets** - Asset loading and management system
|
||||
- **rend** - 3D rendering system with mesh, camera, and material support
|
||||
- **audio** - Sound engine integration
|
||||
- **ui** - User interface rendering system
|
||||
- **papyrus** - Text rendering and UI primitives
|
||||
- **imgui** - Debug UI integration
|
||||
- **physics** - Physics engine integration (Jolt)
|
||||
- **sys** - System utilities and subprocess management
|
||||
|
||||
### Key Directories
|
||||
|
||||
- `engine/` - Core engine modules, each with their own `build.zig`
|
||||
- `projects/` - Sample projects and games
|
||||
- `lib/` - Third-party dependencies and wrappers
|
||||
- `tools/` - Development utilities and scripts
|
||||
- `content/` - Game assets and resources (located at `projects/content/`)
|
||||
- `extras/` - Optional engine extensions
|
||||
|
||||
### Engine Initialization
|
||||
|
||||
The engine uses a spec-based initialization system where modules are conditionally enabled:
|
||||
|
||||
```zig
|
||||
// Programs define which modules to enable
|
||||
sampleGame.setModuleEnabled("imgui", true);
|
||||
sampleGame.setModuleEnabled("physics", true);
|
||||
```
|
||||
|
||||
### Dynamic Modules
|
||||
|
||||
The engine supports dynamic module loading for game-specific code:
|
||||
|
||||
```zig
|
||||
const externGame = sampleGame.addDynamicModule("externGame", b.path("sampleGame/externGame/externGame.zig"));
|
||||
```
|
||||
|
||||
### Shader Pipeline
|
||||
|
||||
Shaders are written in HLSL and compiled to multiple targets:
|
||||
- SPIR-V for Vulkan
|
||||
- MSL for Metal (macOS)
|
||||
- DXIL for DirectX
|
||||
|
||||
The shader compilation system automatically discovers `.hlsl` files in `engine/*/shaders/` directories.
|
||||
|
||||
## Development Notes
|
||||
|
||||
- All memory allocations go through a centralized `MemoryTracker` for leak detection
|
||||
- The engine uses Tracy for profiling when enabled
|
||||
- Lua scripting is integrated for game logic
|
||||
- The build system generates API wrappers automatically for enabled modules
|
||||
- Content directory location is determined by `content.txt` file pointing to `projects/content/`
|
||||
|
||||
## Common Development Tasks
|
||||
|
||||
- Adding new engine modules: Create in `engine/` with `build.zig` and add to `engineDepList`
|
||||
- Creating new projects: Use the project template in `projects/minimal/`
|
||||
- Shader development: Add HLSL files to module `shaders/` directories, run `cookShaders.py`
|
||||
- Asset pipeline: Assets go in `projects/content/` and use `.cook` extensions for processed assets
|
||||
|
||||
## Bug Tracking with git-bug
|
||||
|
||||
This project uses git-bug for distributed bug tracking. Bug data is stored directly in the git repository.
|
||||
|
||||
### Common Commands
|
||||
|
||||
- `git bug bug` - List all bugs
|
||||
- `git bug bug new -t "title" -m "message"` - Create a new bug with title and message
|
||||
- `git bug bug show <id>` - Display bug details
|
||||
- `git bug bug comment <id>` - Add a comment to a bug
|
||||
- `git bug bug status <id>` - Display bug status
|
||||
- `git bug bug label <id> <label>` - Add a label to a bug
|
||||
- `git bug bug status:open` - List only open bugs
|
||||
- `git bug pull` - Pull bug updates from remote
|
||||
- `git bug push` - Push bug updates to remote
|
||||
|
||||
### Workflow
|
||||
|
||||
- Bugs are stored in the repository and sync with `git bug pull`/`git bug push`
|
||||
- Bug IDs can be abbreviated to the first few characters
|
||||
- Use labels to categorize bugs by component (e.g., `core`, `rendering`, `physics`, `build-system`)
|
||||
- Use `--non-interactive` flag for scripting
|
||||
- Keep bug descriptions factual - describe what happens, not speculation about why
|
||||
|
|
@ -0,0 +1,199 @@
|
|||
# Backlog Engine Build Options
|
||||
|
||||
This document provides comprehensive documentation for all `core.BuildOption` flags available in the Backlog game engine.
|
||||
|
||||
## Overview
|
||||
|
||||
Build options in Backlog are boolean flags that control various compilation behaviors and engine features. They are accessed at compile-time using `core.BuildOption("option_name")` and can be set via Zig's build system command line arguments.
|
||||
|
||||
## Usage
|
||||
|
||||
Build options are set when building the project:
|
||||
|
||||
```bash
|
||||
zig build -Doption_name
|
||||
# or
|
||||
zig build -Doption_name=true
|
||||
```
|
||||
|
||||
To disable an option explicitly:
|
||||
```bash
|
||||
zig build -Doption_name=false
|
||||
```
|
||||
|
||||
## Core Engine Build Options
|
||||
|
||||
### Performance & Development Options
|
||||
|
||||
#### `mutex_job_queue`
|
||||
- **Type**: `bool`
|
||||
- **Default**: `false`
|
||||
- **Description**: Temporary test option that reverts to old mutex-based queue behavior in `jobs.zig:JobManager`
|
||||
- **Usage**: `zig build -Dmutex_job_queue`
|
||||
- **Impact**: Changes job system threading behavior for debugging/testing purposes
|
||||
- **Files**: `engine/core/src/jobs.zig:8`
|
||||
|
||||
#### `force_mailbox`
|
||||
- **Type**: `bool`
|
||||
- **Default**: `false`
|
||||
- **Description**: Forces mailbox mode for present mode, unlocking framerate to very high levels
|
||||
- **Usage**: `zig build -Dforce_mailbox`
|
||||
- **Impact**: Disables VSync and allows unlimited framerate rendering
|
||||
- **Warning**: Can cause excessive GPU usage and heat generation
|
||||
|
||||
### Logging Options
|
||||
|
||||
#### `zero_logging`
|
||||
- **Type**: `bool`
|
||||
- **Default**: `false`
|
||||
- **Description**: Disables all logging completely, intended primarily for job dispatch performance testing
|
||||
- **Usage**: `zig build -Dzero_logging`
|
||||
- **Impact**: Removes all log output, improving performance but eliminating debug information
|
||||
- **Files**: `engine/core/src/logging.zig:7`
|
||||
|
||||
#### `slow_logging`
|
||||
- **Type**: `bool`
|
||||
- **Default**: `false`
|
||||
- **Description**: Disables buffered logging, taking a performance hit but providing accurate timing information
|
||||
- **Usage**: `zig build -Dslow_logging`
|
||||
- **Impact**: Each log call is immediately flushed, providing precise timing but reducing performance
|
||||
- **Files**: `engine/core/src/logging.zig:6`
|
||||
|
||||
### Build & Deployment Options
|
||||
|
||||
#### `static_build`
|
||||
- **Type**: `bool`
|
||||
- **Default**: `false`
|
||||
- **Description**: Builds the entire game as a single executable with static linking
|
||||
- **Usage**: `zig build -Dstatic_build`
|
||||
- **Impact**:
|
||||
- Required for Linux builds
|
||||
- Disables dynamic module loading
|
||||
- Creates self-contained executable
|
||||
- **Files**:
|
||||
- `build.zig:190`
|
||||
- `engine/core/src/extern/externModule.zig:139`
|
||||
- `build/generateApi.zig:118`
|
||||
- `engine/rend/src/sgpu/renderer.zig:1126`
|
||||
|
||||
#### `RootDeploymentOnly`
|
||||
- **Type**: `bool`
|
||||
- **Default**: Not explicitly set (appears to default to `false`)
|
||||
- **Description**: Controls whether only root deployment operations are performed
|
||||
- **Usage**: `zig build -DRootDeploymentOnly`
|
||||
- **Impact**: Appears to limit engine initialization to root-level deployment only
|
||||
- **Files**: `engine/main.zig:28`
|
||||
|
||||
### Development & Shader Options
|
||||
|
||||
#### `cookShaders`
|
||||
- **Type**: `bool`
|
||||
- **Default**: `false`
|
||||
- **Description**: Generates shaders and updates `.json` files before running the build
|
||||
- **Usage**: `zig build -DcookShaders`
|
||||
- **Impact**:
|
||||
- Automatically runs `tools/scripts/cookShaders.py`
|
||||
- Compiles HLSL shaders to SPIR-V, MSL, and DXIL formats
|
||||
- Must be run whenever shader files are modified
|
||||
- **Files**: `build.zig:80`
|
||||
|
||||
### Third-Party Integration Options
|
||||
|
||||
#### `tracy` (Tracy Profiler)
|
||||
- **Type**: `bool`
|
||||
- **Default**: `false`
|
||||
- **Description**: Enables Tracy profiler integration for performance analysis
|
||||
- **Usage**: `zig build -Dtracy`
|
||||
- **Impact**:
|
||||
- Adds Tracy profiling instrumentation
|
||||
- Enables real-time performance monitoring
|
||||
- Increases binary size and may affect performance
|
||||
- **Files**: `lib/tracy/build_tracy.zig:37`
|
||||
|
||||
## Build Option Categories
|
||||
|
||||
### By Purpose
|
||||
|
||||
**Performance Testing**:
|
||||
- `mutex_job_queue` - Job system behavior testing
|
||||
- `zero_logging` - Maximum performance logging
|
||||
- `force_mailbox` - Unlimited framerate testing
|
||||
|
||||
**Development & Debugging**:
|
||||
- `slow_logging` - Precise timing information
|
||||
- `tracy` - Performance profiling
|
||||
- `cookShaders` - Shader development workflow
|
||||
|
||||
**Deployment & Distribution**:
|
||||
- `static_build` - Self-contained executable creation
|
||||
- `RootDeploymentOnly` - Deployment scope control
|
||||
|
||||
### By Impact Level
|
||||
|
||||
**High Impact** (affects core engine behavior):
|
||||
- `static_build`
|
||||
- `zero_logging`
|
||||
- `mutex_job_queue`
|
||||
|
||||
**Medium Impact** (affects specific subsystems):
|
||||
- `slow_logging`
|
||||
- `force_mailbox`
|
||||
- `cookShaders`
|
||||
|
||||
**Low Impact** (development/profiling tools):
|
||||
- `tracy`
|
||||
- `RootDeploymentOnly`
|
||||
|
||||
## Common Build Configurations
|
||||
|
||||
### Development Build
|
||||
```bash
|
||||
zig build -Dslow_logging -Dtracy -DcookShaders
|
||||
```
|
||||
|
||||
### Performance Testing Build
|
||||
```bash
|
||||
zig build -Dzero_logging -Dforce_mailbox -Dmutex_job_queue
|
||||
```
|
||||
|
||||
### Production/Release Build
|
||||
```bash
|
||||
zig build -Dstatic_build
|
||||
```
|
||||
|
||||
### Linux Build (Required)
|
||||
```bash
|
||||
zig build -Dstatic_build
|
||||
```
|
||||
|
||||
## Implementation Details
|
||||
|
||||
Build options are implemented through Zig's compile-time options system:
|
||||
|
||||
1. **Definition**: Options are defined in `build.zig` in the `BuildOptions` struct
|
||||
2. **Access**: Runtime code accesses options via `core.BuildOption("option_name")`
|
||||
3. **Compilation**: The `core.BuildOption()` function checks for the option at compile-time
|
||||
4. **Fallback**: If an option is not defined, it defaults to `false`
|
||||
|
||||
The `core.BuildOption()` function implementation:
|
||||
```zig
|
||||
pub fn BuildOption(comptime option: []const u8) bool {
|
||||
if (@hasDecl(@import("root"), "options")) {
|
||||
const r = @import("root").options;
|
||||
if (@hasDecl(r, option)) {
|
||||
return @field(r, option);
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- All build options are boolean flags
|
||||
- Options not explicitly set default to `false`
|
||||
- Some combinations may be incompatible (e.g., `zero_logging` + `slow_logging`)
|
||||
- Linux builds require `static_build=true`
|
||||
- Shader compilation via `cookShaders` should be run after modifying any `.hlsl` files
|
||||
|
|
@ -0,0 +1,152 @@
|
|||
# Backlog Game Engine Overview
|
||||
|
||||
The Backlog Game Engine is a modern, modular game engine built with Zig, designed for cross-platform game development. It provides a comprehensive set of tools and systems for creating interactive 3D applications and games.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **Zig 0.14** or later
|
||||
- **Python 3** for build scripts
|
||||
|
||||
## Running Examples
|
||||
|
||||
The engine comes with several examples to demonstrate its capabilities:
|
||||
|
||||
### Sample Game
|
||||
|
||||
The main example showcasing most engine features including:
|
||||
- 3D rendering with meshes and textures
|
||||
- Physics simulation
|
||||
- ImGui debug interface
|
||||
- Input handling
|
||||
- Dynamic module loading
|
||||
|
||||
**To run the sample game:**
|
||||
|
||||
```bash
|
||||
cd projects
|
||||
zig build run-sampleGame
|
||||
```
|
||||
|
||||
To compile and target linux with cross compiling,
|
||||
|
||||
```
|
||||
zig build -Dtarget=x86_64-linux-gnu
|
||||
```
|
||||
|
||||
The linux build is always compiled as a static executable.
|
||||
|
||||
**Sample Game Controls:**
|
||||
- `WASD` - Move camera
|
||||
- `Mouse` - Look around (toggle menu with `T`)
|
||||
- `E/Q` - Move up/down
|
||||
- `R` - Reload shaders
|
||||
- `Escape` - Exit
|
||||
|
||||
### Minimal Example
|
||||
|
||||
A basic template demonstrating the minimal setup required for a Backlog application:
|
||||
|
||||
```bash
|
||||
cd projects/minimal
|
||||
zig build
|
||||
zig build run-minimal
|
||||
```
|
||||
|
||||
This example shows:
|
||||
- Basic engine initialization
|
||||
- ImGui integration
|
||||
- Simple game loop structure
|
||||
|
||||
## Creating New Projects
|
||||
|
||||
The engine includes a project creation tool to help you get started with new projects quickly.
|
||||
|
||||
### Using the New Project Tool
|
||||
|
||||
1. Navigate to the projects directory:
|
||||
```bash
|
||||
cd projects
|
||||
```
|
||||
|
||||
2. Run the project creator:
|
||||
```bash
|
||||
zig build run-newProject
|
||||
```
|
||||
|
||||
3. Click "Create Project" to generate your new project structure
|
||||
|
||||
### Manual Project Creation
|
||||
|
||||
Alternatively, you can create projects manually by copying the minimal template:
|
||||
|
||||
1. Copy the `projects/minimal` directory:
|
||||
```bash
|
||||
cp -r projects/minimal projects/my-new-project
|
||||
```
|
||||
|
||||
2. Edit `projects/my-new-project/build.zig` and update:
|
||||
- Project name
|
||||
- Root source file path
|
||||
- Enabled modules as needed
|
||||
|
||||
3. Modify `src/main.zig` to implement your game logic
|
||||
|
||||
### Project Structure
|
||||
|
||||
A typical Backlog project includes:
|
||||
|
||||
```
|
||||
my-project/
|
||||
├── build.zig # Build configuration
|
||||
├── build.zig.zon # Dependencies
|
||||
├── src/
|
||||
│ └── main.zig # Main game code
|
||||
├── content/ # Game assets (optional)
|
||||
└── Saved/ # Runtime generated files
|
||||
```
|
||||
|
||||
## Engine Modules
|
||||
|
||||
When creating projects, you can enable these modules:
|
||||
|
||||
- **core** - Essential engine systems (always enabled)
|
||||
- **platform** - Windowing and input (always enabled)
|
||||
- **assets** - Asset loading and management (always enabled)
|
||||
- **rend** - 3D rendering system (always enabled)
|
||||
- **imgui** - Debug UI and developer tools
|
||||
- **audio** - Sound and music playback
|
||||
- **physics** - Physics simulation (Jolt Physics)
|
||||
- **ui** - User interface rendering
|
||||
- **papyrus** - Text rendering and UI primitives
|
||||
- **sys** - System utilities
|
||||
|
||||
## Asset Pipeline
|
||||
|
||||
Assets are placed in the `content/` directory within your project:
|
||||
|
||||
all shaders are cooked to the target system's native format. DXIL for dx12, msl for metal and spv for vulkan.
|
||||
|
||||
These cooked format shaders are placed under content/_shaders
|
||||
|
||||
## Building and Development
|
||||
|
||||
### Building Projects
|
||||
|
||||
From your project directory:
|
||||
```bash
|
||||
zig build # Build project
|
||||
zig build run-[projectname] # Build and run
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
Run tests for the entire engine:
|
||||
```bash
|
||||
zig build test
|
||||
```
|
||||
|
||||
Or test individual modules:
|
||||
```bash
|
||||
cd engine/core
|
||||
zig build test
|
||||
```
|
||||
|
|
@ -38,7 +38,8 @@ pub const EngineObjectDelegate = engineObject.EngineObjectDelegate;
|
|||
pub const FieldInfo = engineObject.FieldInfo;
|
||||
|
||||
pub const jobs = @import("jobs.zig");
|
||||
pub const JobContext = jobs.JobContext;
|
||||
pub const JobContext = jobs.ThreadContext;
|
||||
pub const ThreadContext = jobs.ThreadContext;
|
||||
pub const JobManager = jobs.JobManager;
|
||||
pub const JobWorker = jobs.JobWorker;
|
||||
|
||||
|
|
@ -376,8 +377,30 @@ pub fn shutdown_module(_: std.mem.Allocator) void {
|
|||
return;
|
||||
}
|
||||
|
||||
pub fn parallelJob(capture: anytype, async: bool, maxWorkerCount: ?u32) !void {
|
||||
const z = tracy.ZoneNC(@src(), "parallel Job", 0xCF8774);
|
||||
defer z.End();
|
||||
|
||||
const jobManager = getEngine().jobManager;
|
||||
const requestedWorkers = if (maxWorkerCount) |m| m else jobManager.numWorkers;
|
||||
const baseWorkerCount = @min(requestedWorkers, @max(getEngine().jobManager.numWorkers - 2, 1));
|
||||
|
||||
const workerStart:u32 = if (async) 0 else 1;
|
||||
|
||||
for (workerStart .. baseWorkerCount) |workerId| {
|
||||
try jobManager.newJob(capture, .{ .threadId = @intCast(workerId), .threadCount = baseWorkerCount});
|
||||
}
|
||||
|
||||
// start working on the job locally
|
||||
if (!async) {
|
||||
try jobManager.runLocally(capture, .{.threadId = 0, .threadCount = baseWorkerCount});
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
|
||||
pub fn dispatchJob(capture: anytype) !void {
|
||||
try gEngine.jobManager.newJob(capture);
|
||||
try gEngine.jobManager.newJob(capture, .{.threadId = 0, .threadCount = 1});//0, 1, null);
|
||||
}
|
||||
|
||||
pub fn createObject(comptime T: type, params: engine.NeonObjectParams) !*T {
|
||||
|
|
|
|||
|
|
@ -111,7 +111,7 @@
|
|||
|
||||
pub fn createEntity() !Entity {
|
||||
const rv = Entity{ .handle = try getRegistry().baseSet.createObject(.{}) };
|
||||
core.engine_log("entity created: {d}", .{rv.handle.index});
|
||||
// core.engine_log("entity created: {d}", .{rv.handle.index});
|
||||
return rv;
|
||||
}
|
||||
|
||||
|
|
@ -132,7 +132,7 @@ pub fn destroyAllEntities() void {
|
|||
}
|
||||
|
||||
pub fn destroyEntity(e: Entity) void {
|
||||
core.engine_log("entity destroyed: {d}", .{e.handle.index});
|
||||
// core.engine_log("entity destroyed: {d}", .{e.handle.index});
|
||||
const registry = getRegistry();
|
||||
if (registry.baseSet.get(e.handle)) |entityEntry| {
|
||||
for (entityEntry.containers.items) |ref| {
|
||||
|
|
@ -377,7 +377,7 @@ pub const Entity = struct {
|
|||
}
|
||||
|
||||
pub fn addComponent(self: @This(), comptime Component: type) ?*Component {
|
||||
core.engine_log("adding component: {d} {s}", .{ self.handle.index, @typeName(Component) });
|
||||
// core.engine_log("adding component: {d} {s}", .{ self.handle.index, @typeName(Component) });
|
||||
const rv = Component.BaseContainer.createWithHandleECS(self.handle);
|
||||
|
||||
const list = &getRegistry().baseSet.get(self.handle).?.containers;
|
||||
|
|
|
|||
|
|
@ -230,6 +230,7 @@ pub const Engine = struct {
|
|||
}
|
||||
|
||||
pub fn tick(self: *@This()) !void {
|
||||
// ------------ frame updates ---------
|
||||
tracy.FrameMark();
|
||||
tracy.FrameMarkStart("frame");
|
||||
defer tracy.FrameMarkEnd("frame");
|
||||
|
|
@ -263,7 +264,9 @@ pub const Engine = struct {
|
|||
z2.End();
|
||||
|
||||
self.frameNumber += 1;
|
||||
// -------------------------------------------
|
||||
|
||||
// sync
|
||||
var z3 = tracy.ZoneN(@src(), "platform event updates");
|
||||
if (self.platformProcEventsFunc) |procEventsFn| {
|
||||
try procEventsFn(self.platformCtx, self.frameNumber);
|
||||
|
|
@ -274,12 +277,14 @@ pub const Engine = struct {
|
|||
objectRef.vtable.processEvents.?(objectRef.ptr, self.frameNumber) catch @panic("process event error");
|
||||
}
|
||||
|
||||
// pretick
|
||||
var z4 = tracy.ZoneN(@src(), "pretick event updates");
|
||||
for (self.preTickables.items) |*objectRef| {
|
||||
objectRef.vtable.preTick_func.?(objectRef.ptr, self.deltaTime) catch @panic("pretick event error");
|
||||
}
|
||||
z4.End();
|
||||
|
||||
// tick
|
||||
var z5 = tracy.ZoneN(@src(), "system object ticks");
|
||||
var index: isize = @as(isize, @intCast(self.tickables.items.len)) - 1;
|
||||
while (index >= 0) : (index -= 1) {
|
||||
|
|
@ -293,6 +298,7 @@ pub const Engine = struct {
|
|||
|
||||
const systemsThreadTime: f64 = time.getEngineTime() - newTime;
|
||||
|
||||
// renderer
|
||||
for (self.renderers.items) |*renderer| {
|
||||
var z = tracy.Zone(@src());
|
||||
z.Name(renderer.vtable.typeName);
|
||||
|
|
|
|||
|
|
@ -7,13 +7,18 @@ const ArrayListUnmanaged = std.ArrayListUnmanaged;
|
|||
|
||||
const mutex_job_queue = core.BuildOption("mutex_job_queue");
|
||||
|
||||
pub const TaskInstanceOptions = struct {
|
||||
threadId: u32,
|
||||
threadCount: u32,
|
||||
};
|
||||
|
||||
pub const JobManager = struct {
|
||||
allocator: std.mem.Allocator,
|
||||
|
||||
// need a mutex for the jobQueue... todo later
|
||||
jobQueue: RingQueueU(JobContext),
|
||||
jobQueue: RingQueueU(ThreadContext),
|
||||
mutex: std.Thread.Mutex = .{},
|
||||
jobQueueConcurrent: core.ConcurrentQueueU(JobContext),
|
||||
jobQueueConcurrent: core.ConcurrentQueueU(ThreadContext),
|
||||
workers: []*JobWorker,
|
||||
numCpus: usize,
|
||||
numWorkers: u32 = 0,
|
||||
|
|
@ -24,8 +29,8 @@ pub const JobManager = struct {
|
|||
self.* = JobManager{
|
||||
.allocator = allocator,
|
||||
.numCpus = std.Thread.getCpuCount() catch 4,
|
||||
.jobQueue = RingQueueU(JobContext).init(allocator, 4096) catch unreachable,
|
||||
.jobQueueConcurrent = core.ConcurrentQueueU(JobContext).initCapacity(allocator, 4096) catch unreachable,
|
||||
.jobQueue = RingQueueU(ThreadContext).init(allocator, 4096) catch unreachable,
|
||||
.jobQueueConcurrent = core.ConcurrentQueueU(ThreadContext).initCapacity(allocator, 4096) catch unreachable,
|
||||
.workers = undefined,
|
||||
};
|
||||
|
||||
|
|
@ -35,17 +40,28 @@ pub const JobManager = struct {
|
|||
|
||||
var i: usize = 0;
|
||||
while (i < self.workers.len) : (i += 1) {
|
||||
self.workers[i] = JobWorker.init(self.allocator, i) catch unreachable;
|
||||
self.workers[i].workerId = i;
|
||||
self.workers[i] = JobWorker.init(self.allocator, i, true) catch unreachable;
|
||||
self.workers[i].workerThreadNumber = i;
|
||||
self.workers[i].manager = self;
|
||||
}
|
||||
|
||||
return self;
|
||||
}
|
||||
|
||||
pub fn newJob(self: *@This(), capture: anytype) !void {
|
||||
pub fn runLocally(self: *@This(), capture: anytype, task: TaskInstanceOptions) !void {
|
||||
const Lambda = @TypeOf(capture);
|
||||
const ctx = try JobContext.new(self.allocator, Lambda, capture);
|
||||
const ctx = try ThreadContext.new(self.allocator, Lambda, capture, task);
|
||||
|
||||
//pub fn init(allocator: std.mem.Allocator, workerNumber: usize, detached:bool) !*@This() {
|
||||
const worker = try JobWorker.init(self.allocator, 0xff, false); // todo move these things into a pool of non-detached workers that we can just grab whenever
|
||||
defer worker.deinit();
|
||||
worker.currentJobContext = ctx;
|
||||
worker.run();
|
||||
}
|
||||
|
||||
pub fn newJob(self: *@This(), capture: anytype, task: TaskInstanceOptions) !void {
|
||||
const Lambda = @TypeOf(capture);
|
||||
const ctx = try ThreadContext.new(self.allocator, Lambda, capture, task);
|
||||
|
||||
if (mutex_job_queue) {
|
||||
self.mutex.lock();
|
||||
|
|
@ -91,7 +107,7 @@ pub const JobManager = struct {
|
|||
}
|
||||
|
||||
pub fn clearJobs(self: *@This()) void {
|
||||
var jobCtx: ?JobContext = null;
|
||||
var jobCtx: ?ThreadContext = null;
|
||||
|
||||
if (mutex_job_queue) {
|
||||
self.mutex.lock();
|
||||
|
|
@ -117,15 +133,16 @@ pub const JobManager = struct {
|
|||
|
||||
pub const JobWorker = struct {
|
||||
detached: bool = true, // most threads are detached, completions are handled via callbacks.
|
||||
currentJobContext: ?JobContext = null,
|
||||
workerThread: std.Thread,
|
||||
currentJobContext: ?ThreadContext = null,
|
||||
workerThread: ?std.Thread = null,
|
||||
futex: Atomic(u32) = Atomic(u32).init(0),
|
||||
current: u32 = 0,
|
||||
shouldDie: Atomic(bool) = Atomic(bool).init(false),
|
||||
busy: Atomic(bool) = Atomic(bool).init(false),
|
||||
allocator: std.mem.Allocator,
|
||||
workerId: usize = 0,
|
||||
workerThreadNumber: usize = 0, // identifier for debugging purposes, never changes.
|
||||
manager: ?*JobManager = null,
|
||||
scratchArena: std.heap.ArenaAllocator,
|
||||
|
||||
pub fn wake(self: *JobWorker) void {
|
||||
// this needs to be re-evaluated, it's nice for system performance but
|
||||
|
|
@ -133,15 +150,20 @@ pub const JobWorker = struct {
|
|||
std.Thread.Futex.wake(&self.futex, 1);
|
||||
}
|
||||
|
||||
pub fn init(allocator: std.mem.Allocator, workerNumber: usize) !*@This() {
|
||||
pub fn init(allocator: std.mem.Allocator, workerNumber: usize, detached: bool) !*@This() {
|
||||
const self = try allocator.create(JobWorker);
|
||||
|
||||
self.* = .{
|
||||
.allocator = allocator,
|
||||
.workerThread = try std.Thread.spawn(.{}, @This().workerThreadFunc, .{self}),
|
||||
.workerId = workerNumber,
|
||||
.detached = detached,
|
||||
.workerThreadNumber = workerNumber,
|
||||
.scratchArena = std.heap.ArenaAllocator.init(allocator),
|
||||
};
|
||||
|
||||
if (detached) {
|
||||
self.workerThread = try std.Thread.spawn(.{}, @This().workerThreadFunc, .{self});
|
||||
}
|
||||
|
||||
return self;
|
||||
}
|
||||
|
||||
|
|
@ -149,8 +171,26 @@ pub const JobWorker = struct {
|
|||
return self.busy.load(.acquire);
|
||||
}
|
||||
|
||||
fn run(self: *@This()) void {
|
||||
self.busy.store(true, .seq_cst);
|
||||
|
||||
if(self.manager)|manager|
|
||||
{
|
||||
manager.bump();
|
||||
}
|
||||
|
||||
var ctx = self.currentJobContext.?;
|
||||
ctx.workerInfo = self;
|
||||
ctx.func(ctx.capture, &ctx);
|
||||
self.busy.store(false, .seq_cst);
|
||||
|
||||
ctx.deinit();
|
||||
if (self.scratchArena.reset(.retain_capacity)) {}
|
||||
self.currentJobContext = null;
|
||||
}
|
||||
|
||||
pub fn workerThreadFunc(self: *@This()) void {
|
||||
const printed = std.fmt.allocPrintSentinel(self.allocator, "WorkerThread_{d}", .{self.workerId}, 0) catch unreachable;
|
||||
const printed = std.fmt.allocPrintSentinel(self.allocator, "WorkerThread_{d}", .{self.workerThreadNumber}, 0) catch unreachable;
|
||||
tracy.InitThread();
|
||||
tracy.SetThreadName(@as([*:0]u8, @ptrCast(printed.ptr)));
|
||||
|
||||
|
|
@ -161,12 +201,7 @@ pub const JobWorker = struct {
|
|||
while (!self.shouldDie.load(.acquire)) {
|
||||
if (self.currentJobContext != null) {
|
||||
wakeGrabCount = 0;
|
||||
self.busy.store(true, .seq_cst);
|
||||
var ctx = self.currentJobContext.?;
|
||||
ctx.func(ctx.capture, &ctx);
|
||||
self.busy.store(false, .seq_cst);
|
||||
ctx.deinit();
|
||||
self.currentJobContext = null;
|
||||
self.run();
|
||||
} else {
|
||||
std.Thread.Futex.wait(&self.futex, self.current);
|
||||
}
|
||||
|
|
@ -180,13 +215,6 @@ pub const JobWorker = struct {
|
|||
self.manager.?.mutex.unlock();
|
||||
} else {
|
||||
self.currentJobContext = manager.jobQueueConcurrent.pop();
|
||||
//while (wakeGrabCount > 0) : (wakeGrabCount -= 1) {
|
||||
// self.currentJobContext = manager.jobQueueConcurrent.pop();
|
||||
// if (self.currentJobContext != null) {
|
||||
// break;
|
||||
// }
|
||||
// std.Thread.sleep(1000 * 1000);
|
||||
//}
|
||||
wakeGrabCount = 1;
|
||||
}
|
||||
}
|
||||
|
|
@ -200,28 +228,70 @@ pub const JobWorker = struct {
|
|||
pub fn deinit(self: *@This()) void {
|
||||
self.shouldDie.store(true, .seq_cst);
|
||||
self.wake();
|
||||
self.workerThread.join();
|
||||
if(self.workerThread)|workerThread|
|
||||
{
|
||||
workerThread.join();
|
||||
}
|
||||
|
||||
self.scratchArena.deinit();
|
||||
self.allocator.destroy(self);
|
||||
}
|
||||
};
|
||||
|
||||
pub const JobContext = struct {
|
||||
const Src = std.builtin.SourceLocation;
|
||||
pub fn Barrier(src: Src) type {
|
||||
return struct {
|
||||
pub const Src = src;
|
||||
|
||||
pub var count: u32 = 0;
|
||||
pub var generation: u32 = 0;
|
||||
pub var mutex: std.Thread.Mutex = .{};
|
||||
pub var cv: std.Thread.Condition = .{};
|
||||
|
||||
pub fn sync(threadId: u32, threadCount: u32) !void {
|
||||
_ = threadId;
|
||||
|
||||
mutex.lock();
|
||||
defer mutex.unlock();
|
||||
|
||||
count += 1;
|
||||
|
||||
if (count == threadCount) {
|
||||
count = 0;
|
||||
generation += 1;
|
||||
cv.broadcast();
|
||||
} else {
|
||||
// 2 second timeout
|
||||
try cv.timedWait(&mutex, 1000 * 1000 * 1000 * 2);
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
pub const ThreadContext = struct {
|
||||
const Self = @This();
|
||||
|
||||
funcName: []const u8 = "unnamed",
|
||||
allocator: std.mem.Allocator, //todo, backed arena allocator would be sick for this.
|
||||
func: *const fn (*anyopaque, *JobContext) void, // todo, add an error for job funcs
|
||||
func: *const fn (*anyopaque, *ThreadContext) void, // todo, add an error for job funcs
|
||||
capture: *anyopaque = undefined,
|
||||
threadId: u32,
|
||||
threadIdCount: u32,
|
||||
destroyFunc: *const fn (*anyopaque, std.mem.Allocator) void,
|
||||
workerInfo: ?*JobWorker = null,
|
||||
|
||||
const align8_struct = struct { size: u64 };
|
||||
|
||||
pub fn new(allocator: std.mem.Allocator, comptime CaptureType: type, capture: CaptureType) !JobContext {
|
||||
pub fn new(
|
||||
allocator: std.mem.Allocator,
|
||||
comptime CaptureType: type,
|
||||
capture: CaptureType,
|
||||
task: TaskInstanceOptions,
|
||||
) !ThreadContext {
|
||||
if (!@hasDecl(CaptureType, "func")) {
|
||||
return error.NoValidLambda;
|
||||
}
|
||||
|
||||
const Wrap = struct {
|
||||
pub fn wrappedFunc(pointer: *anyopaque, context: *JobContext) void {
|
||||
pub fn wrappedFunc(pointer: *anyopaque, context: *ThreadContext) void {
|
||||
var ptr = @as(*CaptureType, @ptrCast(@alignCast(pointer)));
|
||||
ptr.func(context);
|
||||
}
|
||||
|
|
@ -234,6 +304,9 @@ pub const JobContext = struct {
|
|||
|
||||
const self = Self{
|
||||
.allocator = allocator,
|
||||
.threadId = task.threadId,
|
||||
.threadIdCount = task.threadCount,
|
||||
.funcName = @typeName(CaptureType),
|
||||
.func = Wrap.wrappedFunc,
|
||||
.destroyFunc = Wrap.wrappedDestroy,
|
||||
.capture = try allocator.create(CaptureType),
|
||||
|
|
@ -243,6 +316,22 @@ pub const JobContext = struct {
|
|||
return self;
|
||||
}
|
||||
|
||||
// helper functions
|
||||
|
||||
pub fn scratch(self: *@This()) std.mem.Allocator {
|
||||
return self.workerInfo.?.scratchArena.allocator();
|
||||
}
|
||||
|
||||
pub fn splitSlice(self: @This(), comptime T: type, slice: []T) struct{ slice: []T, startIndex: usize } {
|
||||
const r = core.p2.splitSliceWork(T, slice, self.threadId, self.threadIdCount);
|
||||
return .{.slice = r.slice, .startIndex = r.startIndex};
|
||||
|
||||
}
|
||||
|
||||
pub fn barrier(self: *@This(), comptime src: std.builtin.SourceLocation) !void {
|
||||
try Barrier(src).sync(self.threadId, self.threadIdCount);
|
||||
}
|
||||
|
||||
pub fn deinit(self: *Self) void {
|
||||
self.destroyFunc(self.capture, self.allocator);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ pub fn build(b: *std.Build) void {
|
|||
|
||||
const luac = b.addLibrary(.{
|
||||
.name = "luac",
|
||||
.linkage = if (static_build) .dynamic else .static,
|
||||
.linkage = if (static_build) .static else .dynamic,
|
||||
.root_module = b.createModule(.{
|
||||
.target = target,
|
||||
.optimize = optimize,
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ pub const getFileExtension = utils.getFileExtension;
|
|||
pub const getBasePath = utils.getBasePath;
|
||||
pub const getFolder = utils.getFolder;
|
||||
pub const getDir = utils.getFolder;
|
||||
pub const splitSliceWork = utils.splitSliceWork;
|
||||
|
||||
pub const BumpArena = @import("structures/bump-arena.zig").BumpArena;
|
||||
|
||||
|
|
|
|||
|
|
@ -171,3 +171,30 @@ pub fn stringStrip(p: []const u8) []const u8 {
|
|||
|
||||
return p[s..e];
|
||||
}
|
||||
|
||||
|
||||
pub fn splitSliceWork(comptime T: type, slice:[]T, id:u32, count:u32 ) struct{ slice: []T, startIndex: usize }
|
||||
{
|
||||
const stride = @divFloor(slice.len, count);
|
||||
const rem = @rem(slice.len, count);
|
||||
|
||||
// let s = stride if index < remainder
|
||||
// let r = stride if index >= remainder
|
||||
//
|
||||
// formula:
|
||||
// x < R; start = s * x; stride = s
|
||||
// x >=R; start = R * s + (x - R) * r; stride = r
|
||||
//
|
||||
// combining...
|
||||
//
|
||||
const s = stride + 1;
|
||||
const r = stride;
|
||||
|
||||
if (id < rem) {
|
||||
const start = s * id;
|
||||
return .{.slice = slice[start .. start + s], .startIndex = start};
|
||||
} else {
|
||||
const start = s * rem + (id - rem) * r;
|
||||
return .{.slice = slice[start .. start + r], .startIndex = start};
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ pub fn build(b: *std.Build) void {
|
|||
|
||||
//const tracy_enabled = b.option(bool, "tracy", "Enables tracy integration") orelse false;
|
||||
|
||||
const tracy_enabled: bool = false; //if (b.graph.env_map.hash_map.get("WITH_TRACY") != null) true else false;
|
||||
const tracy_enabled: bool = true; //if (b.graph.env_map.hash_map.get("WITH_TRACY") != null) true else false;
|
||||
// if (b.graph.env_map.hash_map.get("WITH_TRACY")) |with_tracy| {
|
||||
// tracy_enabled = with_tracy;
|
||||
// }
|
||||
|
|
|
|||
|
|
@ -12,22 +12,34 @@ pub fn create(allocator: std.mem.Allocator) !*@This() {
|
|||
}
|
||||
|
||||
pub fn prepare(self: *@This()) !void {
|
||||
try self.testThing();
|
||||
_ = self;
|
||||
|
||||
for(0..120_000) |i|
|
||||
{
|
||||
_ = i;
|
||||
const e = try core.createEntity();
|
||||
const x = e.addComponent(core.Scene).?;
|
||||
_ = x;
|
||||
}
|
||||
|
||||
// try self.testThing();
|
||||
}
|
||||
|
||||
pub fn tick(self: *@This(), dt: f64) void {
|
||||
// std.Thread.sleep(1000_000);
|
||||
self.timeLeft -= dt;
|
||||
|
||||
if (self.timeLeft < 0)
|
||||
core.exitNow();
|
||||
|
||||
const start = core.getEngineTime();
|
||||
|
||||
core.parallelJob(UpdateWorldTransformsJob{}, false, 1) catch unreachable;
|
||||
|
||||
// wait 10 ms
|
||||
while (core.getEngineTime() - start < 0.010) {
|
||||
std.Thread.sleep(10_000_000);
|
||||
// std.Thread.sleep(10_000_000);
|
||||
}
|
||||
|
||||
if (self.timeLeft < 0)
|
||||
core.exitNow();
|
||||
}
|
||||
|
||||
pub fn deinit(self: *@This()) void {
|
||||
|
|
@ -44,11 +56,9 @@ pub fn Barrier(src: Src) type {
|
|||
pub var mutex: std.Thread.Mutex = .{};
|
||||
pub var cv: std.Thread.Condition = .{};
|
||||
|
||||
pub fn sync(threadId: u32) !void {
|
||||
pub fn sync(threadId: u32, barrierCount: u32) !void {
|
||||
_ = threadId;
|
||||
|
||||
const barrierCount = core.getBarrierCount();
|
||||
|
||||
mutex.lock();
|
||||
defer mutex.unlock();
|
||||
|
||||
|
|
@ -67,25 +77,23 @@ pub fn Barrier(src: Src) type {
|
|||
}
|
||||
|
||||
const MultiJob = struct {
|
||||
pub var WorkGroupData = WorkGroupData.from(@This());
|
||||
|
||||
threadId: u32 = 0,
|
||||
threadCount: u32 = 1,
|
||||
threadName: []const u8,
|
||||
|
||||
pub var cv: std.Thread.Condition = .{};
|
||||
pub var mutex: std.Thread.Mutex = .{};
|
||||
|
||||
pub fn func(ctx: @This(), job: *core.JobContext) void {
|
||||
_ = job;
|
||||
core.tracy.SetThreadName(@ptrCast(ctx.threadName.ptr));
|
||||
ctx.loop() catch {};
|
||||
|
||||
if (core.getEngine().isShuttingDown()) {}
|
||||
|
||||
Barrier(@src()).sync(ctx.threadId, ctx.threadCount) catch unreachable;
|
||||
}
|
||||
|
||||
pub fn loop(ctx: @This()) !void {
|
||||
while (!core.getEngine().isShuttingDown()) {
|
||||
try Barrier(@src()).sync(ctx.threadId);
|
||||
try Barrier(@src()).sync(ctx.threadId, ctx.threadCount);
|
||||
try ctx.tick();
|
||||
if (ctx.threadId == 0) {
|
||||
const z = core.tracy.ZoneN(@src(), "Thread 0 sleep");
|
||||
|
|
@ -100,10 +108,80 @@ const MultiJob = struct {
|
|||
defer z.End();
|
||||
// std.Thread.sleep(100_000 * ctx.threadId);
|
||||
|
||||
try Barrier(@src()).sync(ctx.threadId);
|
||||
try Barrier(@src()).sync(ctx.threadId, ctx.threadCount);
|
||||
}
|
||||
};
|
||||
|
||||
const UpdateWorldTransformsJob = struct {
|
||||
world: ?*anyopaque = null,
|
||||
|
||||
pub fn func(self: @This(), thread: *core.ThreadContext) void {
|
||||
const z = core.tracy.ZoneN(@src(), "Transform Hierarchy - wide");
|
||||
defer z.End();
|
||||
thread.barrier(@src()) catch unreachable;
|
||||
|
||||
var outputs: std.ArrayList(core.math.Transform) = .{};
|
||||
var outputList: std.ArrayList(usize) = .{};
|
||||
|
||||
self.updateTransformsHierarchy(thread, &outputs, &outputList) catch |err| switch (err) {
|
||||
error.OutOfMemory => {
|
||||
unreachable;
|
||||
},
|
||||
// else => {
|
||||
// thread.abort(@src(), "unknown error", err);
|
||||
// return;
|
||||
// },
|
||||
};
|
||||
|
||||
// sync results of output List
|
||||
thread.barrier(@src()) catch unreachable;
|
||||
|
||||
// write out all results, there likely is no issue with false sharing as transforms are exactly 64 bytes
|
||||
// const dense = self.world.denseScenes();
|
||||
// for (outputList.items) |i| {
|
||||
// // dense[i].transform = outputs.items[i];
|
||||
// }
|
||||
}
|
||||
|
||||
pub fn updateTransformsHierarchy(self: @This(), thread: *core.ThreadContext, outputs: *std.ArrayList(core.math.Transform), outputList: *std.ArrayList(usize)) !void {
|
||||
_ = outputList;
|
||||
_ = self;
|
||||
//const dense = self.world.denseScenes();
|
||||
const densePosRot = core.Scene.SceneObjectContainer.denseItems(.posRot);
|
||||
|
||||
// everything allocated with thread.scratch is blown away when the thread is complete
|
||||
try outputs.resize(thread.scratch(), densePosRot.len);
|
||||
|
||||
// scan and mark all root nodes for update
|
||||
const split = thread.splitSlice(core.scene.ScenePosRot, densePosRot);
|
||||
|
||||
const locals:[]core.math.Transform = try thread.scratch().alloc(core.math.Transform, split.slice.len);
|
||||
|
||||
for (split.slice, 0..) |posRot, i| {
|
||||
const index = split.startIndex + i;
|
||||
_ = index;
|
||||
|
||||
locals[i] = posRot.toTransform();
|
||||
}
|
||||
|
||||
//for (split.slice, 0..) |repr, i| {
|
||||
//}
|
||||
}
|
||||
};
|
||||
|
||||
// new idea that im thinking i want to do...
|
||||
//
|
||||
// game is now responsible for scheduling order of work
|
||||
//
|
||||
// eg. prepare_game now has to return a struct that defines a list of phases. eg. the default list looks like,
|
||||
//
|
||||
// engine.setEngineTickPhases(&.{
|
||||
// core.tick,
|
||||
// // ecs.tick
|
||||
// });
|
||||
//
|
||||
// core. comes with a bunch of prebuilt phases.
|
||||
|
||||
pub fn testThing(self: *@This()) !void {
|
||||
_ = self;
|
||||
const workerCount = 16;
|
||||
|
|
@ -112,8 +190,14 @@ pub fn testThing(self: *@This()) !void {
|
|||
|
||||
for (0..workerCount) |i| {
|
||||
const name = try std.fmt.allocPrintSentinel(std.heap.c_allocator, "MultiJob{d}", .{i}, 0);
|
||||
try core.dispatchJob(MultiJob{ .threadName = name, .threadId = @intCast(i) });
|
||||
try core.dispatchJob(MultiJob{ .threadName = name, .threadId = @intCast(i), .threadCount = workerCount });
|
||||
}
|
||||
|
||||
// workerCountHint=0
|
||||
// async==false
|
||||
// will block until the job is complete, the active thread will also construct a threadContext that picks up one of the parallel tasks (todo implement workstealing)
|
||||
// try core.dispatchMulti(UpdateWorldTransformsJob{ .world = self.mainWorld }, null, false); // null == use max workers,
|
||||
|
||||
}
|
||||
|
||||
pub fn main() anyerror!void {
|
||||
|
|
|
|||
|
|
@ -0,0 +1,2 @@
|
|||
Saved/*
|
||||
zig-out
|
||||
Loading…
Reference in New Issue