LLM one-shots a deadlock

deadlock-llm-fix

// LLM one-shots a deadlock - https://github.com/nem0/LumixEngine/issues/1534
// Claude Opus 4.5.

User: When I add a dynamic point light with shadows to a scene with a few objects and move the camera and objects for a while, the editor freezes. All worker threads are in this callstack
 	[External Code]	
>	studio.exe!Lumix::ConditionVariable::sleep(Lumix::Mutex & mutex) Line 54	C++
 	studio.exe!Lumix::Thread::sleep(Lumix::Mutex & mutex) Line 149	C++
 	studio.exe!Lumix::jobs::popWork(Lumix::jobs::Work & work, Lumix::jobs::WorkerTask * worker) Line 401	C++
 	studio.exe!Lumix::jobs::manage(void * data) Line 561	C++
 	[External Code]	
What's the issue?

GitHub Copilot: Read [](file:///c%3A/projects/LumixEngine/src/core/job_system.cpp#380-380), lines 380 to 600

Searched for text `point light shadow`, no results

Searched for text `pointLightShadow`, no results

Searched for text `PointLightShadow`, no results

Searched for regex `point.*shadow|shadow.*point`, 15 results

Searched for text `bakeShadow`, 3 results

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#3350-3350), lines 3350 to 3500

Let me look at the `bakeShadow` function more closely: 

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#820-820), lines 820 to 950

Now I need to look at the `cull` function and the `renderBucket` function to understand the job system usage: 

Searched for regex `u32 cull\(` (`**/pipeline.cpp`), 1 result

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#990-990), lines 990 to 1150

Now let me look at `createSortKeys` to understand the jobs system usage and the potential deadlock: 

Searched for text `createSortKeys` (`**/pipeline.cpp`), 3 results

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#3770-3770), lines 3770 to 3950

Now let me look at what happens after `createSortKeys` and understand the synchronization: 

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#1025-1025), lines 1025 to 1070

Let me understand the job system's signal mechanism better, and see how `pushJob` works: 

Searched for text `pushJob` (`**/renderer.cpp`), no results

Searched for files matching `**/renderer.cpp`, 1 match

Searched for text `void pushJob`, 2 results

Read [](file:///c%3A/projects/LumixEngine/src/renderer/renderer.h#110-110), lines 110 to 180

Now let me understand the `setupJob` function: 

Searched for text `setupJob` (`**/renderer.cpp`), 1 result

Read [](file:///c%3A/projects/LumixEngine/src/renderer/renderer.cpp#990-990), lines 990 to 1030

Now I see that `setupJob` uses `jobs::run` with a signal `m_cpu_frame->setup_done`. Let me look at how the frame synchronization works: 

Searched for text `setup_done` (`**/renderer.cpp`), 4 results

Read [](file:///c%3A/projects/LumixEngine/src/renderer/renderer.cpp#1140-1140), lines 1140 to 1270

Now let me understand what happens in the `bakeShadow` function when it's inside the "fill clusters" job. Let me look at the entire flow more carefully: 

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#3410-3410), lines 3410 to 3440

I see now! The `bakeShadow` calls are happening **before** the `pushJob("fill clusters")`, not inside it. Let me look more closely at what `bakeShadow` does, particularly the `cull` and `renderBucket` calls inside it: 

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#830-830), lines 830 to 920

Now I see the pattern. `bakeShadow` calls `cull()` which in turn calls `m_renderer.pushJob()`. Let me look again at the `cull` function to see what waits are set up: 

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#985-985), lines 985 to 1060

I see the issue now. In the `cull` function:

1. `m_sort_keys_group.beginJob()` is called
2. Inside the "prepare view" job, `m_sort_keys_group.endJob()` is called
3. Later in the same job, `m_sort_keys_group.wait()` is called

When `bakeShadow` is called for 6 faces (omnidirectional light), it calls `cull` 6 times. Each call to `cull` does `m_sort_keys_group.beginJob()`, but only one `endJob()` can happen per job. Let me look at the `JobGroup` implementation: 

Searched for text `struct JobGroup`, no results

Searched for text `m_sort_keys_group`, 9 results

Searched for text `struct JobsGroup`, 1 result

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#275-275), lines 275 to 350

Now I understand the issue. Looking at the code flow:

1. `bakeShadow` is called from line 3421/3424 in `fillClusters`
2. `bakeShadow` calls `cull` for each of the 6 faces (omnidirectional light)  
3. Each `cull` call does `m_sort_keys_group.beginJob()` which increments the counter
4. Then it pushes a "prepare view" job that will call `m_sort_keys_group.endJob()` and later `m_sort_keys_group.wait()`

The problem is that `m_sort_keys_group` is a single shared instance for the entire pipeline. When multiple `cull` calls are made (e.g., 6 times for an omnidirectional light), the `beginJob` counter is incremented 6 times, but the `wait` in any of the "prepare view" jobs will wait for ALL of them to complete.

But wait - the more critical issue is that if a "prepare view" job waits on `m_sort_keys_group.wait()`, and there are other "prepare view" jobs that haven't run yet (which would call `endJob()`), this creates a potential deadlock if all worker threads are blocked waiting.

Let me look at how jobs are scheduled and whether there's a finite pool of workers: 

Continue to iterate?
[object Object]

User: @agent Continue: "Continue to iterate?"

GitHub Copilot: Read [](file:///c%3A/projects/LumixEngine/src/core/job_system.cpp#1-1), lines 1 to 100

Searched for text `getWorkersCount` (`**/job_system.cpp`), 2 results

Read [](file:///c%3A/projects/LumixEngine/src/core/job_system.cpp#615-615), lines 615 to 650

Now let me look at how `jobs::wait` works, especially to understand if fibers are involved: 

Searched for text `void wait` (`**/job_system.cpp`), 3 results

Read [](file:///c%3A/projects/LumixEngine/src/core/job_system.cpp#655-655), lines 655 to 720

Now I understand the job system better. When `wait` is called, the current fiber is parked and the worker switches to another fiber to do other work. This is fiber-based job scheduling, which should handle waiting inside jobs well.

Let me look more closely at the fiber pool: 

Searched for text `m_fiber_pool` (`**/job_system.cpp`), 3 results

Searched for text `popFreeFiber` (`**/job_system.cpp`), 4 results

Read [](file:///c%3A/projects/LumixEngine/src/core/job_system.cpp#240-240), lines 240 to 290

I see that `popFreeFiber` asserts if there's no free fiber. Let me check if there's ever a case where all fibers are exhausted: 

Read [](file:///c%3A/projects/LumixEngine/src/core/job_system.cpp#285-285), lines 285 to 330

Now let me look at the issue from a different angle. The freeze happens when all worker threads are sleeping waiting for work. Let me look at what happens with the `m_poses_done` signal: 

Searched for text `m_poses_done` (`**/pipeline.cpp`), 3 results

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#3745-3745), lines 3745 to 3780

Now let me look at the key synchronization points. The issue might be related to how the "prepare view" job waits for `m_sort_keys_group` and `m_poses_done`. Let me trace the execution more carefully.

Let me understand the flow when `bakeShadow` is called:

1. `fillClusters` calls `bakeShadow` for dynamic point lights
2. `bakeShadow` (for omnidirectional light) calls `cull` 6 times (for each face)
3. Each `cull` call:
   - Creates a new view
   - Calls `m_sort_keys_group.beginJob()` 
   - Pushes a "prepare view" job that will eventually call `m_sort_keys_group.endJob()` and later `m_sort_keys_group.wait()`

Wait, I think I see the issue! In `cull`, there's:
- `m_sort_keys_group.beginJob()` (line 1029) 
- Then "prepare view" job calls `m_sort_keys_group.endJob()` (line 1041) and then `m_sort_keys_group.wait()` (line 1047)

But `m_sort_keys_group` is a shared member! So when `bakeShadow` is called for 6 faces:
1. First `cull` calls `beginJob()` → counter = 1 (and turns signal red)
2. Second `cull` calls `beginJob()` → counter = 2
3. ... continues until counter = 6
4. First "prepare view" job runs, calls `endJob()` → counter = 5
5. First "prepare view" job then calls `wait()` but counter is still 5, so it blocks waiting for all 6 views to finish!

But wait, the "prepare view" jobs also have their own internal work via `createSortKeys` which uses `jobs::runOnWorkers`. Let me look at this: 

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#3800-3800), lines 3800 to 3820

Searched for text `runOnWorkers` (`**/job_system.cpp`), no results

Searched for text `void runOnWorkers`, 2 results

Read [](file:///c%3A/projects/LumixEngine/src/core/job_system.h#115-115), lines 115 to 145

Now I understand the issue better. `runOnWorkers` runs on all workers and **waits for all to complete** synchronously.

Inside the "prepare view" job, `createSortKeys` is called which internally calls `jobs::runOnWorkers`. So if a "prepare view" job is waiting at `m_sort_keys_group.wait()` while another "prepare view" hasn't been picked up yet, we have a potential deadlock.

Let me trace again with concrete flow:

1. `fillClusters` is called on the main thread
2. `bakeShadow` is called, and for omnidirectional light it runs a loop 6 times
3. For each iteration of the loop, `cull` is called which pushes a "prepare view" job
4. Each "prepare view" job:
   - Runs on any worker
   - Calls `createSortKeys` which uses `jobs::runOnWorkers` (blocks until all workers process)
   - Then waits on `m_sort_keys_group` 

The critical issue is that `jobs::runOnWorkers` requires **all workers** to run the job, but if some workers are already blocked waiting on `m_sort_keys_group.wait()`, there won't be enough free workers!

Wait, but the fiber system should handle this. When `wait` is called, the fiber yields and the worker picks up new work.

Let me re-read the flow for the edge case...

Actually, let me look at whether the issue is about the `m_sort_keys_group` shared state between multiple `cull` calls: 

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#275-275), lines 275 to 310

I see the issue now! In `JobsGroup`:

- `begin()` sets `counter = 1` and turns signal red
- `beginJob()` just increments the counter
- `endJob()` decrements and turns green **only if counter becomes 0 (after dec, it's 1 before dec)**
- `wait()` waits for the signal to be green

The problem is in `cull`:
- Line 1029: `m_sort_keys_group.beginJob()` is called (not `begin()`)!

But `beginJob()` just increments the counter. It doesn't set `counter = 1` or turn the signal red! Let me trace what happens:

1. First `cull`: `beginJob()` increments counter (0 → 1), but **signal is still green!**
   - Later, `endJob()` decrements (1 → 0), turning signal green (it was already green)
   - `wait()` on green signal returns immediately

2. Second `cull`: Same thing... 

Wait, there's an issue. `beginJob()` doesn't turn the signal red when counter goes 0 → 1. Let me check if `begin()` is ever called: 

Searched for regex `m_sort_keys_group\.(begin|end)\(` (`**/pipeline.cpp`), 3 results

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#1570-1570), lines 1570 to 1660

Now I understand the design better. `renderMain()` calls `m_sort_keys_group.begin()` and `m_sort_keys_group.end()` to bracket the entire main render pass. The `cull` function calls `beginJob()` and `endJob()` to track individual "prepare view" jobs.

But the problem is: `cull` is being called from `bakeShadow` which is called **before** `renderMain` is called in `fillClusters`!

Let me look at where `fillClusters` is called: 

Searched for text `fillClusters` (`**/pipeline.cpp`), 2 results

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#1800-1800), lines 1800 to 1850

I see now - `fillClusters` is called **before** `renderMain`. So when `bakeShadow` is called inside `fillClusters`, `m_sort_keys_group.begin()` has **not** been called yet.

This means when `cull` calls `m_sort_keys_group.beginJob()`:
- If counter was 0 and signal was green, `beginJob()` just increments counter to 1
- But it doesn't turn the signal red!

When `endJob()` is called:
- Counter goes 1 → 0
- Since `counter.dec() == 1` (the value before decrement was 1), it turns green... but it was already green!

When `wait()` is called:
- Signal is green, so it should return immediately

But wait, that should work then. Let me look at `beginJob()` more carefully:

```cpp
void beginJob() {
    counter.inc();
}
```

And `endJob()`:
```cpp
void endJob() {
    if (counter.dec() == 1) {
        jobs::turnGreen(&signal);
    } 
}
```

The condition `counter.dec() == 1` means the value **before decrement** was 1 (so after decrement it's 0). This should work correctly...

But wait! Let me look at the signal management. When `cull` calls `beginJob()`, if the counter goes from 0 to 1, **the signal should turn red** but it doesn't! Let me think about the race condition...

Actually, let me look at what happens when `renderMain` is called after `fillClusters`:

1. `fillClusters` calls `bakeShadow` which calls `cull` multiple times
2. Each `cull` call does `beginJob()` incrementing counter
3. Each corresponding "prepare view" job does `endJob()` decrementing counter
4. Then `renderMain` calls `m_sort_keys_group.begin()` which sets `counter = 1` and turns signal red!

The issue is: if a "prepare view" job from `bakeShadow` hasn't finished yet when `renderMain().begin()` is called, the `counter = 1` assignment will **reset the counter**, potentially corrupting the state!

Actually, let me trace more carefully:

1. `bakeShadow` is called for omnidirectional light
2. For i=0: `cull` is called, `beginJob()` → counter becomes 1
3. For i=0: `renderBucket` is called, which pushes a "render bucket" job that waits on `view->ready`
4. For i=1: `cull` is called, `beginJob()` → counter becomes 2
5. ...
6. After all 6 iterations, counter = 6
7. Later: "prepare view" job for i=0 runs, calls `endJob()` → counter becomes 5, signal still red
8. ...

Wait, but when is the signal turned red initially? `beginJob()` doesn't turn it red!

Looking at the code again:

```cpp
void beginJob() {
    counter.inc();
}
```

If the initial state is counter=0, signal=green, then after `beginJob()`, counter=1, signal=green.

When the "prepare view" job calls:
1. `endJob()` → counter goes 1→0, since `dec() == 1`, it calls `turnGreen()` (which is a no-op since already green)
2. `wait()` → signal is green, returns immediately

This should work... but the freeze still happens. Let me look at what happens when multiple `bakeShadow` calls interleave:

Actually, I realize the issue might be with the comment on line 1046:
```cpp
// Wait for all createSortKeys jobs to finish, ensuring no more pose processing jobs will be created.
m_sort_keys_group.wait();
```

The `createSortKeys` function uses `jobs::runOnWorkers` internally. Let me trace:

1. "prepare view" job starts
2. Calls `createSortKeys(*view_ptr)` which internally calls `jobs::runOnWorkers([&](){ ... })`
3. `runOnWorkers` runs the lambda on all workers and **blocks** until all complete
4. After `createSortKeys` returns, `m_sort_keys_group.endJob()` is called
5. Then `m_sort_keys_group.wait()` is called

The issue is: `runOnWorkers` blocks the current fiber waiting for all workers. But if multiple "prepare view" jobs are running, and they all call `runOnWorkers`, they compete for the worker threads.

Actually, I think the fiber system should handle this. When `runOnWorkers` waits, the fiber yields.

Let me look at the actual issue from a different angle. The callstack shows workers are sleeping waiting for work. This means there's no work in any queue. Let me look at what could cause all work to be consumed without producing more: 

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#1025-1025), lines 1025 to 1060

I found the bug! Look at the logic in the "prepare view" job:

```cpp
if (view_ptr->renderables && (view_ptr->renderables->header.count != 0 || view_ptr->renderables->header.next)) {
    createSortKeys(*view_ptr);
    view_ptr->renderables->free(m_renderer.getEngine().getPageAllocator());
}
m_sort_keys_group.endJob();  // This is OUTSIDE the if!
view_ptr->sorter.pack();
    
if (!view_ptr->sorter.keys.empty()) {
    radixSort(...);
    m_sort_keys_group.wait();  // This is INSIDE the second if!
    jobs::wait(&m_poses_done);
    createCommands(*view_ptr);
}
```

The problem is:
1. `m_sort_keys_group.endJob()` is called unconditionally
2. `m_sort_keys_group.wait()` is called **only if** `!view_ptr->sorter.keys.empty()`

When `bakeShadow` is called and creates 6 views (for omnidirectional light), the counter starts at some value. After all 6 `cull` calls, counter = 6 (assuming it started at 0).

Now, if some of these views have empty renderables (nothing visible from that face):
- "prepare view" job runs
- `endJob()` is called → counter decrements
- But `wait()` is NOT called because `sorter.keys.empty()` is true!
- This view's job is done

For views that DO have renderables:
- `endJob()` is called → counter decrements  
- `wait()` IS called
- But it will wait forever if there are still pending jobs that haven't called `endJob()` yet!

Wait, but the issue is actually the opposite. Let me think again...

If counter = 6 after 6 `cull` calls:
- All 6 "prepare view" jobs call `endJob()` regardless of whether they have renderables
- Counter goes 6 → 5 → 4 → 3 → 2 → 1 → 0
- When it hits 0, signal turns green

But then some jobs call `wait()` and some don't. The jobs that call `wait()` wait for the signal to turn green, which happens when counter = 0.

Actually, the logic seems correct. Let me look more carefully at `beginJob()`:

```cpp
void beginJob() {
    counter.inc();
}
```

This doesn't turn the signal red! So if the signal was already green and counter was 0:
1. First `cull`: `beginJob()` → counter = 1, signal still green
2. Second `cull`: `beginJob()` → counter = 2, signal still green
3. ... counter = 6, signal still green
4. First "prepare view" runs, `endJob()` → counter = 5, signal green (no change because counter > 1)
5. ... until counter = 1
6. Last `endJob()` → counter = 0, signal turns green (was already green, no-op)
7. `wait()` returns immediately because signal is green

This should work... unless there's a race condition where `renderMain` starts before all `bakeShadow` views are processed!

Let me look at the execution order again:

```cpp
if (only_2d) {
    render2DOnly();
}
else {
    fillClusters(stream, getMainCamera());  // This calls bakeShadow which calls cull
    renderMain();  // This calls m_sort_keys_group.begin() setting counter = 1!
}
```

The issue is:
1. `fillClusters` calls `bakeShadow` which calls `cull` 6 times, incrementing counter to 6
2. 6 "prepare view" jobs are pushed to the job queue
3. `fillClusters` returns
4. `renderMain` is called
5. `renderMain` calls `m_sort_keys_group.begin()` which does `counter = 1` and `turnRed(&signal)`!

Now counter = 1 (not 6!), and signal is red. But there are still 6 pending "prepare view" jobs from `bakeShadow`!

When those jobs run:
- They call `endJob()` → counter goes 1 → 0 → **-1** → **-2** (underflow!)
- The signal state becomes corrupted!

Or worse, if one "prepare view" job runs before `renderMain`:
- It calls `endJob()` → counter goes 6 → 5
- Then `renderMain` calls `begin()` → counter = 1, signal red
- Other "prepare view" jobs call `endJob()` → counter goes 1 → 0 → -1 → ...

This is the bug! `m_sort_keys_group.begin()` in `renderMain` corrupts the state set up by `bakeShadow`'s `cull` calls.

The fix should be to NOT call `m_sort_keys_group.begin()` and `m_sort_keys_group.end()` in `renderMain`, OR to ensure that `bakeShadow` uses a separate `JobsGroup` or doesn't use `cull` (which uses the shared `m_sort_keys_group`).

Let me verify by looking at how `bakeShadow` works: 

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#920-920), lines 920 to 990

Yes, I can confirm that `bakeShadow` calls `cull` (line 879 and 943) which uses `m_sort_keys_group.beginJob()`.

The fix should be to ensure that `bakeShadow`'s use of `cull` doesn't interfere with `renderMain`'s use of `m_sort_keys_group`. 

The cleanest fix would be to:
1. Move the `m_sort_keys_group.begin()` call to **before** `fillClusters` 
2. Or use a separate tracking mechanism for `bakeShadow`

Let me look at the intended design - the `begin()`/`end()` pair should bracket all `beginJob()`/`endJob()` pairs. Looking at line 1581:

```cpp
void renderMain() {
    m_sort_keys_group.begin();
    ...
```

This suggests that the design intended all `cull` calls to happen AFTER `begin()` is called. But `fillClusters` is called BEFORE `renderMain`, breaking this assumption.

The fix should move `m_sort_keys_group.begin()` to happen before `fillClusters`: 

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#1780-1780), lines 1780 to 1825

Now I see the code structure. The fix should move `m_sort_keys_group.begin()` to before `fillClusters`. Let me look at `renderMain()` again to understand what else uses the group: 

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#1580-1580), lines 1580 to 1620

Now I understand the full picture. The `m_sort_keys_group.begin()` is called at the start of `renderMain()` and `end()` is called at the end. The issue is that `fillClusters` (called before `renderMain`) also uses `cull` which calls `beginJob()`.

The fix is to move `m_sort_keys_group.begin()` to before `fillClusters` is called. Let me implement this: 

Using "Multi-Replace String in Files"


```

```
Now let me verify the changes and build to make sure there are no compile errors: 

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#1810-1810), lines 1810 to 1825

Read [](file:///c%3A/projects/LumixEngine/src/renderer/pipeline.cpp#1578-1578), lines 1578 to 1590

The changes look correct. Now let me build to verify there are no compile errors: 

Ran `Build All (Debug)`

Made changes.