bug(perf): GPU timestamp readback returns stale samples at GPU-bound frame rates #1076

Closed
opened 2026-11-07 03:28:22 +00:00 by SakulFlee · 8 comments
Owner

Summary

The GPU timing columns in the Windows capture are not updating. Every GPU stage has only 6 distinct values over 180 one-second samples:

  • gpu_shadow: 1.413 for rows 1-92, then 1.135 for rows 96-179
  • gpu_models: 2.293 for rows 1-92, then 1.277 for rows 96-179
  • gpu_volumetrics: 2.414 for rows 1-92, then 2.114 for rows 96-179
  • All GPU channels change together only at rows 93-95

Real GPU time does not stay constant to 0.001 ms for 92 seconds. The values change exactly during 2.4-2.8 ms iced stalls, when the CPU finally gives the readback time to complete.

Likely cause

Renderer::render writes timestamps into one of two staging buffers, then immediately polls the previous buffer, maps it, and unmaps it even if the map did not finish. When the GPU frame time is close to the CPU frame time, the previous buffer is still in use and the map never completes; prev_gpu_ns stays at the last successful sample. timestamp_readback_ms is ~0.006 ms, confirming that no wait occurs.

Impact

  • The Windows GPU columns are unusable for per-second analysis.
  • Linux captures show the same long constant runs; cross-platform GPU comparisons are suspect.
  • It blocks the Windows volumetric investigation and the Android hybrid regression analysis.

Fix direction

  • Do not unmap a pending map; wait for the correct submission or fence before mapping.
  • Use at least three staging buffers, or a dedicated readback thread/callback.
  • Add a stale-sample counter or health flag and surface it in the CSV/debug UI.
  • Add a test that records N frames with a simulated GPU delay and asserts that GPU samples change.

Acceptance criteria

  • GPU columns vary frame-to-frame on Windows/Linux when timestamp queries are available.
  • No multi-second constant runs unless the workload is truly constant.
  • The Windows performance CSV is re-captured and attached.
## Summary The GPU timing columns in the Windows capture are not updating. Every GPU stage has only 6 distinct values over 180 one-second samples: - `gpu_shadow`: `1.413` for rows 1-92, then `1.135` for rows 96-179 - `gpu_models`: `2.293` for rows 1-92, then `1.277` for rows 96-179 - `gpu_volumetrics`: `2.414` for rows 1-92, then `2.114` for rows 96-179 - All GPU channels change together only at rows 93-95 Real GPU time does not stay constant to 0.001 ms for 92 seconds. The values change exactly during 2.4-2.8 ms iced stalls, when the CPU finally gives the readback time to complete. ## Likely cause `Renderer::render` writes timestamps into one of two staging buffers, then immediately polls the previous buffer, maps it, and unmaps it even if the map did not finish. When the GPU frame time is close to the CPU frame time, the previous buffer is still in use and the map never completes; `prev_gpu_ns` stays at the last successful sample. `timestamp_readback_ms` is ~0.006 ms, confirming that no wait occurs. ## Impact - The Windows GPU columns are unusable for per-second analysis. - Linux captures show the same long constant runs; cross-platform GPU comparisons are suspect. - It blocks the Windows volumetric investigation and the Android hybrid regression analysis. ## Fix direction - Do not unmap a pending map; wait for the correct submission or fence before mapping. - Use at least three staging buffers, or a dedicated readback thread/callback. - Add a stale-sample counter or health flag and surface it in the CSV/debug UI. - Add a test that records N frames with a simulated GPU delay and asserts that GPU samples change. ## Acceptance criteria - GPU columns vary frame-to-frame on Windows/Linux when timestamp queries are available. - No multi-second constant runs unless the workload is truly constant. - The Windows performance CSV is re-captured and attached.
Author
Owner

Android phone data extends this beyond desktop:

  • Rows 1-48 report gpu_scene_total around 1 ms with almost no variation.
  • Rows 49-64 report every GPU stage as exactly 0.000.
  • At the same time, present is ~89 ms and FPS is ~8.

A real GPU frame cannot be 1 ms while present blocks for 89 ms. This should be treated as stale/unavailable GPU data, not as evidence that the GPU is fast.

Important distinction: the overall frame timing is still real. update + surface_acq + total matches cdt, and cdt matches FPS. The reporting problem is limited to the GPU-stage columns; it does not invalidate the measured 8 FPS.

Android phone data extends this beyond desktop: - Rows 1-48 report `gpu_scene_total` around 1 ms with almost no variation. - Rows 49-64 report every GPU stage as exactly `0.000`. - At the same time, `present` is ~89 ms and FPS is ~8. A real GPU frame cannot be 1 ms while present blocks for 89 ms. This should be treated as stale/unavailable GPU data, not as evidence that the GPU is fast. Important distinction: the overall frame timing is still real. `update + surface_acq + total` matches `cdt`, and `cdt` matches FPS. The reporting problem is limited to the GPU-stage columns; it does not invalidate the measured 8 FPS.
Author
Owner

Tablet data is even clearer:

  • gpu_scene_total is 3.149 for every non-zero row.
  • gpu_volumetrics is 4.876 for every non-zero row.
  • All GPU stages have only two distinct values: zero and one constant.
  • present is ~342 ms and FPS is ~3.

A real 3 FPS frame cannot have 3.1 ms of scene GPU work and 4.9 ms of volumetrics. These are stale samples, not a fast GPU. Android should be included in the scope of this fix, and unavailable/stale GPU data should be marked explicitly rather than reported as a plausible number.

Tablet data is even clearer: - `gpu_scene_total` is `3.149` for every non-zero row. - `gpu_volumetrics` is `4.876` for every non-zero row. - All GPU stages have only two distinct values: zero and one constant. - `present` is ~342 ms and FPS is ~3. A real 3 FPS frame cannot have 3.1 ms of scene GPU work and 4.9 ms of volumetrics. These are stale samples, not a fast GPU. Android should be included in the scope of this fix, and unavailable/stale GPU data should be marked explicitly rather than reported as a plausible number.
Author
Owner

The new capture shows why this still matters:

  • gpu_complete (independent queue-completion callback): ~195 ms median
  • gpu_scene_total: constant ~1.09 ms
  • gpu_volumetrics: constant ~0.15 ms
  • present: ~86 ms
  • frame period: ~107 ms

The pass timestamps are still frozen/incorrect while the independent GPU completion measurement shows a nearly two-frame queue backlog. The stale columns cannot identify which pass is responsible, so this issue remains a blocker for GPU-pass attribution.

The new capture shows why this still matters: - `gpu_complete` (independent queue-completion callback): ~195 ms median - `gpu_scene_total`: constant ~1.09 ms - `gpu_volumetrics`: constant ~0.15 ms - `present`: ~86 ms - frame period: ~107 ms The pass timestamps are still frozen/incorrect while the independent GPU completion measurement shows a nearly two-frame queue backlog. The stale columns cannot identify which pass is responsible, so this issue remains a blocker for GPU-pass attribution.
Author
Owner

Real tablet data reinforces this: timestamp-query features are reported as available on the Adreno 810, but gpu_scene_total is all zero while gpu_complete is ~567 ms and the frame is ~358 ms. The pass timestamps are unusable on both Android GPUs.

Real tablet data reinforces this: timestamp-query features are reported as available on the Adreno 810, but `gpu_scene_total` is all zero while `gpu_complete` is ~567 ms and the frame is ~358 ms. The pass timestamps are unusable on both Android GPUs.
Author
Owner

Implemented on feat/perf-timings, commit dd8a5479:

  • GPU timestamp readback now uses a four-slot asynchronous ring.
  • Pending maps are never unmapped or overwritten.
  • Completed slots are polled without blocking; stale generations are ignored.
  • Timestamp values are multiplied by Queue::get_timestamp_period().
  • If all slots are busy, timestamp capture is skipped for that frame instead of corrupting the buffer.

This is not yet hardware-validated. Please re-capture the tablet or phone with the new commit; the GPU pass columns (gpu_scene_total, gpu_models, gpu_volumetrics, gpu_tonemap, gpu_aa) should now update instead of staying zero/frozen. Keep this issue open until the Android capture confirms it.

Implemented on `feat/perf-timings`, commit `dd8a5479`: - GPU timestamp readback now uses a four-slot asynchronous ring. - Pending maps are never unmapped or overwritten. - Completed slots are polled without blocking; stale generations are ignored. - Timestamp values are multiplied by `Queue::get_timestamp_period()`. - If all slots are busy, timestamp capture is skipped for that frame instead of corrupting the buffer. This is not yet hardware-validated. Please re-capture the tablet or phone with the new commit; the GPU pass columns (`gpu_scene_total`, `gpu_models`, `gpu_volumetrics`, `gpu_tonemap`, `gpu_aa`) should now update instead of staying zero/frozen. Keep this issue open until the Android capture confirms it.
Author
Owner

The timestamp fix is working on the emulator (llvmpipe, x86_64, 320x640). The GPU pass columns are no longer zero:

  • gpu_scene_total: ~89 ms
  • gpu_shadow: ~30 ms
  • gpu_models: ~57 ms
  • gpu_volumetrics: ~130 ms
  • gpu_tonemap: ~14 ms
  • gpu_aa: ~8 ms
  • gpu_complete: ~245 ms
  • frame: ~272 ms

This confirms the readback ring itself works. The values are from software llvmpipe, so they are not tablet performance numbers, but the pass breakdown is now usable. Need a phone/tablet capture with the new build for hardware validation; keeping this open until then.

The timestamp fix is working on the emulator (llvmpipe, x86_64, 320x640). The GPU pass columns are no longer zero: - `gpu_scene_total`: ~89 ms - `gpu_shadow`: ~30 ms - `gpu_models`: ~57 ms - `gpu_volumetrics`: ~130 ms - `gpu_tonemap`: ~14 ms - `gpu_aa`: ~8 ms - `gpu_complete`: ~245 ms - frame: ~272 ms This confirms the readback ring itself works. The values are from software llvmpipe, so they are not tablet performance numbers, but the pass breakdown is now usable. Need a phone/tablet capture with the new build for hardware validation; keeping this open until then.
Author
Owner

Timestamp readback is now confirmed on a real device: Mali-G720 phone, 1220 x 2582, ~8.2 FPS. The pass values are nonzero and the sum matches the frame:

  • gpu_scene_total: ~93 ms
  • gpu_models: ~91 ms
  • gpu_volumetrics: ~14 ms
  • gpu_tonemap: ~7 ms
  • gpu_aa: ~1.6 ms
  • frame: ~122 ms

The old failure mode (all-zero/frozen GPU columns) is gone on Mali. I still want the tablet/Adreno confirmation before closing this issue.

Timestamp readback is now confirmed on a real device: Mali-G720 phone, 1220 x 2582, ~8.2 FPS. The pass values are nonzero and the sum matches the frame: - `gpu_scene_total`: ~93 ms - `gpu_models`: ~91 ms - `gpu_volumetrics`: ~14 ms - `gpu_tonemap`: ~7 ms - `gpu_aa`: ~1.6 ms - frame: ~122 ms The old failure mode (all-zero/frozen GPU columns) is gone on Mali. I still want the tablet/Adreno confirmation before closing this issue.
Author
Owner

The four-slot readback ring and queue timestamp period are working on the Android Mali-G720 capture at 5be0937d: 66 usable rows, GPU values vary per frame, and the split subpasses sum exactly to gpu_scene_total (77.88 ms). The phone no longer shows the long constant runs described for Windows.

Keeping this open for the remaining acceptance item: the Windows capture and, if available, a fresh tablet/Adreno capture.

The four-slot readback ring and queue timestamp period are working on the Android Mali-G720 capture at `5be0937d`: 66 usable rows, GPU values vary per frame, and the split subpasses sum exactly to `gpu_scene_total` (77.88 ms). The phone no longer shows the long constant runs described for Windows. Keeping this open for the remaining acceptance item: the Windows capture and, if available, a fresh tablet/Adreno capture.
Sign in to join this conversation.
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
SakulFlee/Orbital#1076
No description provided.