perf(android): all-in-one Android scene is GPU-bound at ~20 FPS on Mali-G720 #1073
Labels
No labels
Context: Bug
Context: Enhancements
Platform: Android
Platform: Linux
Platform: Web
Platform: Windows
Platform: iOS
Platform: macOS
Release
Target: CI
Target: CLI
Target: Dependency
Target: Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
SakulFlee/Orbital#1073
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Regression data
The regression is not Mali-specific: Adreno drops by roughly 20-27x as well. The hybrid renderer is the common change.
Capture notes
SHADER_F16(#1074), so its captures are scene-only.Blockers
Investigation
Acceptance criteria
perf(android): investigate ~9 FPS on Mali-G720 with hybrid rendererto perf(android): hybrid renderer drops Mali-G720 to ~9 FPS and Adreno 810 to ~3 FPSPhone CSV attached above. The steady-state result is clearly present-bound, not label/SDF-bound and not primarily iced:
cdtmedian 120.4 mssurface_acqmedian 0.61 msupdate13.62 ms (ui_events13.26 ms)scene_submit6.16 msoverlay_layers5.94 ms (iced_draw1.03 ms,iced_submit4.78 ms)total106.47 mspresentmedian 88.85 ms, mean 83.7 msupdate + surface_acq + totalis 120.7 ms and matches the 120.4 ms CDT, so the low FPS is real.presentalone caps the phone at roughly 11 FPS; the remaining CPU work brings it down to ~8.The GPU columns cannot currently explain it:
gpu_scene_totalreports ~1 ms, then all GPU stages become zero after row 48 whilepresentstays around 89 ms. That is inconsistent with a real 1 ms GPU frame and is tracked in #1076. The real GPU work must be at least on the order of the present wait.This also means:
Suggested A/B next:
Tablet CSV attached. It confirms that iced is not the primary cause: the tablet has no iced at all and still runs at ~2-3 FPS.
Stable medians (rows 3+):
cdt: 368.0 msupdate: 0.49 msui_events: 0.045 msoverlay_layers: 0.047 msiced_draw/iced_submit: 0.000 / 0.000surface_acq: 0.52 msrealize: 4.54 msscene_encode: 2.94 msscene_submit: 16.22 mstotal: 367.71 mspresent: 342.33 msupdate + surface_acq + totalis 368.7 ms and matches the 368.0 ms CDT.presentis 93% oftotaland 93% of the full frame. The tablet is present-bound in the same way as the phone, just much worse.Phone vs tablet:
presentupdateiced_submitThe GPU columns are again unusable:
gpu_scene_totalstays at 3.149 ms andgpu_volumetricsat 4.876 ms for 91 rows, whilepresentis 342 ms. That is the stale readback in #1076.Conclusion: both Android devices point to a real present/GPU stall, not labels, iced, or surface acquisition. The tablet is the cleanest data point because iced is completely absent.
Next useful tests:
A practical blocker for the next A/B runs: on the phone the Rendering panel is off-screen, and the mobile "Debug" toggle can become unreachable when an off-screen panel still counts as visible. That is tracked in #1082. Fixing the small-screen layout/recovery path is needed before we can reliably toggle hybrid, volumetrics, AA, and render-scale settings on the device.
Minimal phone CSV attached. This changes the interpretation significantly.
With the debug UI hidden (rows 1-6):
vsync_enabled: true, so 60 is the cap)iced_draw: ~0.05 msiced_submit: ~0.96 msoverlay_layers: ~1.05 msupdate: ~0.25 mspresent: ~0.8-2.7 msgpu_scene_total: ~0.14 mstotal: ~9 msWith the debug UI open (rows 50-84):
iced_draw: ~10.4 msiced_submit: ~4.8 msoverlay_layers: ~15.3 msui_events: ~5.4 msupdate: ~5.6 mspresent: ~0.76 msgpu_scene_total: ~0.12 mstotal: ~19.7 msSo the minimal example is not GPU- or present-bound at all. It is iced/debug-UI-bound: opening the UI adds roughly 18-20 ms per frame and drops it from the 60 FPS cap to ~40 FPS.
This means:
presentplateau (89/342 ms) is not a generic Android present problem. The same phone can present a trivial scene in under 1 ms.This also strongly supports the small-screen tabbed layout in #1082 and broadening #1079 from idle submit to active panel cost.
New branch capture attached: 589 rows over ~635 s, stable ~9.3 FPS.
Stable rows (UI panels hidden for 554 of 584 stable samples; only
World2D|Orbital UI|UI|Debugvisible):surface_acq: 0.42 msupdate: 0.44 ms medianscene_submit: 12.90 ms medianpresent: 85.89 ms mediantotal: 106.50 ms medianframe_gap: 1.40 msgpu_complete: 195.54 ms mediangpu_scene_total: 1.09 ms (still stale/constant)iced_draw/iced_submit: 0.04 / 0.76 msupdate + surface_acq + total + frame_limiter + frame_gapmatchescdt(~108.8 ms). The debug UI is not running in the stable segment.The key new signal is
gpu_complete: ~195 ms, about 1.8× the 107 ms frame period, and it correlates 0.988 withpresent. With a queue depth of two frames, that corresponds to roughly ~97 ms of GPU work per frame, i.e. the frame period. This points to a heavily backed-up GPU queue, not a CPU or surface-acquisition bottleneck.Caveat:
gpu_completeis a queue-completion latency, not a per-frame GPU duration; it includes prior queued frames. So it does not yet tell us which GPU pass is responsible.gpu_scene_total/gpu_volumetricsare still stale (1.09/0.15 ms), so pass-level attribution still needs #1076 or a GPU profiler.Phone/tablet confirmation and the
performance_meta.jsonsidecar are still needed. The presence of iced and the layer names suggests the phone, but the metadata will confirm resolution, refresh, present mode and frame latency.Phone capture attached with metadata. Key numbers:
AutoNoVsync(supported:Mailbox,Fifo), vsync off, desired frame latency 2Stable UI-hidden rows (39 samples, before the panels were opened):
cdt: ~108.8 mssurface_acq: 0.43 msupdate: 0.50 msscene_submit: 6.8 ms medianpresent: 93.3 ms meantotal: 106.9 ms meangpu_complete: 202.3 ms meangpu_scene_total: ~0.2-1.1 ms; still staleWith the debug panels open (28 samples):
update: 14.1 msiced_submit: 6.6 ms meanpresent: 95.2 msgpu_complete: 222.8 mstotal: 118.0 msSo the UI costs frames, but the baseline is still ~9 FPS with the UI hidden.
gpu_completeremains ~1.8–1.9× the frame period, consistent with the GPU queue being the limiter.The phone orientation change is not represented in the CSV: the metadata sidecar is only the startup snapshot and there are no per-row width/height/orientation/refresh fields. That is now a concrete requirement under #1083.
The attached tablet files are not a tablet run: the tablet metadata reports the same phone model, Mali-G720, 1220 x 2582 and OS as the phone, and the tablet CSV is byte-identical to the earlier phone capture. A real tablet capture is still needed.
Real tablet capture confirmed: Adreno 810, 2560 x 1600, scale factor 2.0, Android 16 on 2509ARPBDG, no iced (
Orbital UIhidden, iced draw/submit = 0).Short but stable run (17 usable samples, ~3 FPS):
cdt: 358.6 ms mediansurface_acq: 0.41 msupdate: 0.49 msscene_submit: 14.3 ms medianpresent: 290.8 ms mean / 327.2 ms mediantotal: 351.7 ms mean / 352.9 ms mediangpu_complete: 566.7 ms mean / 678.4 ms mediangpu_scene_total: 0.0 msiced_draw/iced_submit: 0 / 0The phone/tablet comparison:
presentgpu_completegpu_complete / frameBoth devices show ~1.7-1.9x frame-period GPU completion latency, consistent with the configured two-frame queue. The tablet has no iced at all, so the ~3 FPS is not UI-related. This is strong evidence that both Android devices are GPU/queue-bound.
The tablet pass timestamps are unusable (
gpu_scene_totalis all zero), so pass attribution still needs #1076 or a GPU profiler. Resolution/refresh still need normalization (#1083).Real tablet native vs quarter-resolution comparison (same Adreno 810, no iced, UI hidden):
presentgpu_completesurface_acqupdatescene_submitQuarter resolution is 4x fewer pixels and the frame/GPU completion times improved by roughly 3.3-3.4x. That is strong evidence for a pixel/fill-rate/bandwidth-bound workload on the tablet.
This also confirms the earlier tablet files were mislabelled; the attached native/quarter files are the real tablet runs.
The phone baseline at 1220x2582 is ~9 FPS; the tablet at 1280x800 reaches ~8.3 FPS. The tablet's GPU is substantially weaker, but resolution scaling can make it much more usable. The tablet remains GPU/queue-bound at both resolutions (
gpu_complete / frame ~1.8-1.9), so adaptive render scale is a valid mitigation but not a complete fix.The Perfetto trace confirms the tablet is running at a 48 Hz active display mode (supported 120/90/60/50/48/30), while winit reports refresh 0. Thermal status is 0 with GPU around 40 C, so thermal throttling is not the cause.
gpu_counter_trackis empty and the native app has no HWUI frame timeline (gfxinforeports 0 frames), so the trace still cannot identify the GPU pass.Minimal tablet data attached. Assuming this is the minimal scene (metadata shows 4 entities, 2 models, 1 light), it is not equally bad at native resolution:
present: ~11 msgpu_complete: ~42 msupdate: ~0.55 msscene_submit: ~9.6 msscene_encode: ~3.2 msiced: 0gpu_scene_total: 0 (old timestamp bug)Compare with AiO native on the same tablet: ~2.6 FPS / ~372 ms frame. Minimal is roughly 12x faster.
This means scene/render workload complexity absolutely does matter. The earlier assumption that minimal was equally bad is not supported by this capture. The tablet can render the trivial scene at ~33 FPS at 2560x1600; AiO at the same resolution is ~2.6 FPS.
If you meant a different minimal run, please send that CSV/meta too. This one points back at the AiO GPU workload rather than a generic Android present/swapchain limit.
Emulator run with the timestamp fix is useful. It is llvmpipe/CPU, not tablet hardware, but the pass breakdown is now nonzero:
gpu_scene_total: ~89 msgpu_shadow: ~30 msgpu_skybox: ~2.6 msgpu_models: ~57 msgpu_volumetrics: ~130 msgpu_tonemap: ~14 msgpu_aa: ~8 msgpu_complete: ~245 mspresent: ~1 msSo on this software renderer, volumetrics is the largest pass, followed by models and shadows.
presentis negligible, and CPU update is ~0.17 ms. This is evidence that the hybrid scene GPU workload is the problem, not present/swapchain, and it finally gives us pass attribution.The emulator resolution is only 320x640; we need the tablet/phone run on the new build to see whether volumetrics is also dominant on Adreno/Mali. If it is, the regression is likely in the volumetric/hybrid path rather than Android presentation.
Phone capture with the fixed GPU timestamp readback (Mali-G720, 1220 x 2582, UI hidden, ~8.2 FPS):
gpu_scene_totalgpu_shadowgpu_skyboxgpu_modelsgpu_volumetricsgpu_tonemapgpu_aagpu_completepresentThe pass sum is roughly the frame time, and CPU is not the bottleneck (
update~0.5 ms,scene_submit~7.4 ms). The frame is GPU-bound.The largest bucket is
gpu_models, which currently contains G-buffer, light culling, deferred lighting, forward opaque models and transparent models. That bucket is ~90 ms; volumetrics is only ~14 ms on the phone. So unlike llvmpipe, the phone hotspot is the scene/deferred/model group, not primarily volumetrics.The timestamps are now nonzero and coherent, so #1076 is fixed on this device. The next step is to split
gpu_modelsinto its subpasses (G-buffer / light culling / deferred lighting / models / transparent) to identify the exact cost.Phone capture with the Damaged Helmet loaded:
gpu_scene_total: ~82 msgpu_shadow: ~5.2 msgpu_skybox: ~1.1 msgpu_models: ~75.9 msgpu_volumetrics: ~12.6 msgpu_tonemap: ~6.1 msgpu_aa: ~1.6 msgpu_complete: ~196 msThe helmet loads and raises shadow cost (1.1 -> 5.2 ms), but the dominant bucket is still
gpu_modelsat ~76 ms. The missing helmet was not the cause of the low FPS.I added subpass timestamps on
feat/perf-timings(commit5be0937d):gpu_gbuffergpu_light_cullinggpu_deferredgpu_modelsnow means forward opaque modelsgpu_transparentgpu_scene_totalremains the aggregateRebuild and capture the tablet/phone again; the ~76-90 ms bucket should now split into the actual deferred/hybrid cost.
Fresh pass-level capture from
feat/perf-timings@5be0937don the phone (Mali-G720 MC7, 1220x2712, 17 lights, 8 shadow casters, deferred, SMAA High, volumetrics Low):gpu_scene_totalgpu_deferredgpu_gbuffergpu_light_cullinggpu_models(forward opaque)gpu_transparentgpu_shadowgpu_skyboxgpu_volumetricsgpu_tonemapgpu_aaThe subpass sum matches
gpu_scene_totalexactly (77.88 ms). Deferred lighting is 94.6% of the scene total, so the oldgpu_modelsbucket was overwhelmingly deferred lighting, not G-buffer geometry.The previous black screen was not the timestamp code: the generated test project was stale and spawned the 8 SDF billboard labels during
Module::setupbecause it lacked upstream commit1182fd90(the Android SDF guard).generate_sdf/SdfAtlas::build_atlasran for ~35.6 billion inner iterations on the main thread before the first frame. The local project source was synced to the canonical all-in-one example; the timestamp build then rendered. The engine checkout remains clean at5be0937d.Follow-up optimization issue: #1085.
Controlled phone A/B at native resolution (Mali-G720, 1220x2582, same scene/settings) changes the conclusion from “deferred pass” to “shared per-fragment lighting/shadow work”:
gpu_deferred73.61 msgpu_deferred33.15 msgpu_deferred4.01 msApproximate decomposition:
All four variants had exact subpass sums. The forward path is only ~6.8 ms faster than deferred, so fixing the deferred pass alone cannot recover the regression. The remaining work is in the shared per-fragment light/shadow path.
Follow-ups:
One important caveat on the original before/after comparison: switching the current build to the forward path gives 10.38 FPS, not ~130 FPS. The current forward path is only 8.7% faster than deferred, so the 130 FPS “before hybrid” number is not reproducible by simply disabling deferred now.
Before treating #1073 as a hybrid-only regression, verify the old 130 FPS capture on the same scene, resolution, AA, volumetrics, and light/shadow set (or on the pre-hybrid commit). The measured current bottleneck is shared per-fragment lighting/shadow work in both paths.
Root cause candidate from the batch: #1088. The light “range” packed into
params.zisintensity / 0.01; the all-in-one intensities of 30–60 produce radii of ~55–77 world units. That makes every point/spot light global, so tiled culling is all-inclusive and most light/shadow work is evaluated everywhere.This is consistent with:
Next: temporary fixed range (e.g. 10 units) A/B before implementing the explicit range API.
Root cause fixed in
b7086621: point/spot range is now explicit (default 10 world units), glTF range is supported, and the light range is no longer derived from intensity. A smooth range window replaces the hard cutoff.The temporary fixed-range test reduced
gpu_deferredfrom 73.83 to 21.26 ms at radius 10 and raised FPS from 9.50 to 20.00. Rebuild fromb7086621and capture the default-range scene; then re-run the culling and shadow experiments now that the tile lists should be sparse.perf(android): hybrid renderer drops Mali-G720 to ~9 FPS and Adreno 810 to ~3 FPSto perf(android): all-in-one Android scene is GPU-bound at ~20 FPS on Mali-G720b7086621verified on the phone with the restored project (default range 10):b7086621gpu_deferredgpu_scene_totalCulling now does real work (forced full loop: 24.85 ms deferred vs 21.24 tiled), the 8 shadow casters now cost ~4.4 ms instead of ~40.5 ms, and the hard light circles are gone with the smooth range window.
The original “130 FPS before hybrid” number predates the renderer rework and is not a valid comparison target. This issue now tracks the current all-in-one Android performance baseline rather than a hybrid-vs-forward regression.
Tablet A/B on the
b7086621build (Adreno 810, 2560x1600, same project/source):5be0937db7086621gpu_scene_totalgpu_deferredgpu_volumetricsgpu_light_cullingThe range fix works on both vendors. The tablet is now dominated by volumetrics (~162 ms, ~74% of the frame), not deferred lighting or shadows. Follow-up #1089 covers the Adreno volumetric investigation and the tablet light-culling net-benefit check.
Volumetric split and first mobile tier are in:
7060c5bfaddsgpu_volumetric_inject,gpu_volumetric_accumulate,gpu_volumetric_compositeplus the existing aggregate.3233c0b5addsVolumetricQuality::Mobile(128x80x64 at 2560x1600), uses it on Android in the all-in-one example, and switches light tiles from 16px to 8px.Tablet baseline split before the tier: inject ~93 ms, accumulate ~45–59 ms, composite ~26 ms, total ~160–178 ms. The new tier should bring the tablet near the earlier V3 custom result (~34 ms volumetrics), with tile 8 reducing the culling pass from ~13 ms to ~7 ms.
Sync the copied all-in-one source before rebuilding, then recapture tablet and phone.
Mobile tier verified: tablet 4.47 → 11.27 FPS (frame 224 → 88 ms), phone 19.94 → 30.41 FPS. Tablet volumetrics 164.08 → 34.71 ms. Composite is now the common volumetric bottleneck (21.1 ms tablet / 9.16 ms phone), so
de1d9883reduces the Mobile composite from 32 to 16 steps.Remaining tablet split: composite, inject, accumulate; next capture will confirm the step reduction and check for banding.
Mobile tier + 16-step composite verified at
de1d9883:Tablet volumetrics 164.08 → 24.01 ms; phone 12.67 → 5.42 ms. The tablet non-volumetric floor is ~16.8 FPS / 59.3 ms, so remaining volumetric headroom is limited. Remaining tablet split: inject ~11.1 ms, composite ~10.4 ms, accumulate ~2.5 ms.
Further engine optimization is optional; visual range tuning is the next practical step.
Merged as
7d67a8cfvia PR #1093.Final state of the branch:
VolumetricQuality::Mobile: 48 depth slices, 8 composite steps, reduced X/Y grid.Measured final baseline (native resolution, full visuals):
The tablet is now below 30 FPS because of the native 2560×1600 resolution and post chain, not the light-range/volumetric bug. Resolution scaling is tracked separately in #1083.
Closing this root issue; the remaining post-processing and native-resolution work belongs to the follow-up issues.