ci: smart test selection and PR test reports (#839) #890

Merged
SakulFlee merged 3 commits from sakulflee/smart-testing into main 2026-10-22 17:25:30 +00:00 AGit
Owner

Summary

Implements smart testing for pull requests (#839):

  • Runs Rust tests only for the crates directly changed by a PR.
  • Always runs the complete test suite for PRs targeting release/**, manual workflow runs, and release tags.
  • Uses cargo-nextest with a CI profile that emits JUnit XML.
  • Publishes/updates a PR comment with per-crate results, a pass/fail pie chart, a slowest-crates chart, and a failure graph.
  • Runs doctests separately because nextest does not support them.
  • Falls back to the full workspace suite for workspace-level, asset, submodule, or unknown changes.
  • Skips tests for documentation-only PRs; lint already verifies compilation for examples.

No test-report artifacts are stored. The JUnit XML is only used to build the PR comment.

Refs #839

## Summary Implements smart testing for pull requests (#839): - Runs Rust tests only for the crates directly changed by a PR. - Always runs the complete test suite for PRs targeting `release/**`, manual workflow runs, and release tags. - Uses `cargo-nextest` with a CI profile that emits JUnit XML. - Publishes/updates a PR comment with per-crate results, a pass/fail pie chart, a slowest-crates chart, and a failure graph. - Runs doctests separately because nextest does not support them. - Falls back to the full workspace suite for workspace-level, asset, submodule, or unknown changes. - Skips tests for documentation-only PRs; lint already verifies compilation for examples. No test-report artifacts are stored. The JUnit XML is only used to build the PR comment. Refs #839
ci: run changed crate tests and publish PR test reports (#839)
Some checks failed
Lint / lint (pull_request) Successful in 3m35s
PR / test (pull_request) Failing after 21m22s
5a5bf3a189

Smart test results

Result: FAIL

Scope: full workspace

Reason: workspace-level change: .config/nextest.toml

Elapsed: 20m 40s

Workflow run

Overall: 544 tests, 533 passed, 1 failed, 10 skipped

Nextest: 409 tests, 409 passed, 0 failed, 0 skipped
Doctests: 135 tests, 124 passed, 1 failed, 10 skipped

Outcome charts are split between nextest and doctests. The timing chart covers nextest only.

Crate Nextest Passed Failed Skipped Doctests Time
iced 17 17 0 0 18/18 0.10s
iced_beacon 3 3 0 0 - 0.02s
iced_core 4 4 0 0 17/17 0.02s
iced_futures 0 0 0 0 1/2 (1 failed) 0.00s
iced_graphics 3 3 0 0 - 0.15s
iced_highlighter 4 4 0 0 - 0.03s
iced_runtime 2 2 0 0 3/3 0.01s
iced_test 0 0 0 0 4/4 0.00s
iced_widget 46 46 0 0 80/80 2.77s
orbital 1 1 0 0 - 0.43s
orbital_2d 29 29 0 0 0/1 0.17s
orbital_app 9 9 0 0 0/1 0.05s
orbital_camera 4 4 0 0 - 0.02s
orbital_core 2 2 0 0 - 0.01s
orbital_debug_render 0 0 0 0 0/1 0.00s
orbital_debug_ui 9 9 0 0 0/1 0.05s
orbital_ecs 109 109 0 0 0/2 0.65s
orbital_ecs_bridge 18 18 0 0 - 0.10s
orbital_file_manager 5 5 0 0 - 0.05s
orbital_iced 7 7 0 0 - 0.05s
orbital_importer_gltf 13 13 0 0 - 0.15s
orbital_input 10 10 0 0 - 0.05s
orbital_light 6 6 0 0 - 0.03s
orbital_macros 0 0 0 0 0/1 0.00s
orbital_math 4 4 0 0 - 0.02s
orbital_procgeo 2 2 0 0 - 0.01s
orbital_resources 18 18 0 0 - 0.12s
orbital_shader_core 0 0 0 0 1/1 0.00s
orbital_shader_preprocessor 12 12 0 0 - 0.06s
orbital_shadow 3 3 0 0 - 0.02s
orbital_text 14 14 0 0 0/1 0.36s
orbital_texture 9 9 0 0 - 0.05s
orbital_touch_ui 1 1 0 0 0/1 0.01s
orbital_ui 32 32 0 0 0/1 0.17s
orbital_world_environment 13 13 0 0 - 0.07s
pie showData
    title Nextest outcomes
    "Passed" : 409
pie showData
    title Doctest outcomes
    "Passed" : 124
    "Failed" : 1
    "Skipped" : 10
xychart-beta
    title "Slowest crates (nextest seconds)"
    x-axis ["iced_widget", "orbital_ecs", "orbital", "orbital_text", "orbital_ui", "orbital_2d", "orbital_importer_gltf", "iced_graphics"]
    y-axis "Seconds" 0 --> 2.8
    bar [2.77, 0.65, 0.43, 0.36, 0.17, 0.17, 0.15, 0.15]

Failed tests

flowchart LR
  n0["iced_futures"]:::fail
  n1["iced_futures (doctests)"]:::fail
  n2["Crates/orbital-iced/futures/src/subscription.rs - subscription::Subscr"]:::fail
  n0 --> n1 --> n2
  classDef fail fill:#d73a49,color:#ffffff,stroke:#ffffff
Failure details

iced_futures::Crates/orbital-iced/futures/src/subscription.rs - subscription::Subscription (line 94)

error[E0432]: unresolved import `iced_futures::backend::default::time::every`
  --> Crates/orbital-iced/futures/src/subscription.rs:98:17
   |
98 |         pub use iced_futures::backend::default::time::every;
   |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^-----
   |                                                       |
   |                                                       no `every` in `backend::native::thread_pool::time`

error: aborting due to 1 previous error

For more information about this error, try `rustc --explain E0432`.
Couldn't compile the test.
<!-- orbital-ci-test-results --> ## Smart test results **Result:** FAIL **Scope:** full workspace **Reason:** workspace-level change: .config/nextest.toml **Elapsed:** 20m 40s [Workflow run](https://forgejo.sakul-flee.de/SakulFlee/Orbital/actions/runs/433) **Overall:** 544 tests, 533 passed, 1 failed, 10 skipped **Nextest:** 409 tests, 409 passed, 0 failed, 0 skipped **Doctests:** 135 tests, 124 passed, 1 failed, 10 skipped _Outcome charts are split between nextest and doctests. The timing chart covers nextest only._ | Crate | Nextest | Passed | Failed | Skipped | Doctests | Time | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | iced | 17 | 17 | 0 | 0 | 18/18 | 0.10s | | iced_beacon | 3 | 3 | 0 | 0 | - | 0.02s | | iced_core | 4 | 4 | 0 | 0 | 17/17 | 0.02s | | iced_futures | 0 | 0 | 0 | 0 | 1/2 (1 failed) | 0.00s | | iced_graphics | 3 | 3 | 0 | 0 | - | 0.15s | | iced_highlighter | 4 | 4 | 0 | 0 | - | 0.03s | | iced_runtime | 2 | 2 | 0 | 0 | 3/3 | 0.01s | | iced_test | 0 | 0 | 0 | 0 | 4/4 | 0.00s | | iced_widget | 46 | 46 | 0 | 0 | 80/80 | 2.77s | | orbital | 1 | 1 | 0 | 0 | - | 0.43s | | orbital_2d | 29 | 29 | 0 | 0 | 0/1 | 0.17s | | orbital_app | 9 | 9 | 0 | 0 | 0/1 | 0.05s | | orbital_camera | 4 | 4 | 0 | 0 | - | 0.02s | | orbital_core | 2 | 2 | 0 | 0 | - | 0.01s | | orbital_debug_render | 0 | 0 | 0 | 0 | 0/1 | 0.00s | | orbital_debug_ui | 9 | 9 | 0 | 0 | 0/1 | 0.05s | | orbital_ecs | 109 | 109 | 0 | 0 | 0/2 | 0.65s | | orbital_ecs_bridge | 18 | 18 | 0 | 0 | - | 0.10s | | orbital_file_manager | 5 | 5 | 0 | 0 | - | 0.05s | | orbital_iced | 7 | 7 | 0 | 0 | - | 0.05s | | orbital_importer_gltf | 13 | 13 | 0 | 0 | - | 0.15s | | orbital_input | 10 | 10 | 0 | 0 | - | 0.05s | | orbital_light | 6 | 6 | 0 | 0 | - | 0.03s | | orbital_macros | 0 | 0 | 0 | 0 | 0/1 | 0.00s | | orbital_math | 4 | 4 | 0 | 0 | - | 0.02s | | orbital_procgeo | 2 | 2 | 0 | 0 | - | 0.01s | | orbital_resources | 18 | 18 | 0 | 0 | - | 0.12s | | orbital_shader_core | 0 | 0 | 0 | 0 | 1/1 | 0.00s | | orbital_shader_preprocessor | 12 | 12 | 0 | 0 | - | 0.06s | | orbital_shadow | 3 | 3 | 0 | 0 | - | 0.02s | | orbital_text | 14 | 14 | 0 | 0 | 0/1 | 0.36s | | orbital_texture | 9 | 9 | 0 | 0 | - | 0.05s | | orbital_touch_ui | 1 | 1 | 0 | 0 | 0/1 | 0.01s | | orbital_ui | 32 | 32 | 0 | 0 | 0/1 | 0.17s | | orbital_world_environment | 13 | 13 | 0 | 0 | - | 0.07s | ```mermaid pie showData title Nextest outcomes "Passed" : 409 ``` ```mermaid pie showData title Doctest outcomes "Passed" : 124 "Failed" : 1 "Skipped" : 10 ``` ```mermaid xychart-beta title "Slowest crates (nextest seconds)" x-axis ["iced_widget", "orbital_ecs", "orbital", "orbital_text", "orbital_ui", "orbital_2d", "orbital_importer_gltf", "iced_graphics"] y-axis "Seconds" 0 --> 2.8 bar [2.77, 0.65, 0.43, 0.36, 0.17, 0.17, 0.15, 0.15] ``` ### Failed tests ```mermaid flowchart LR n0["iced_futures"]:::fail n1["iced_futures (doctests)"]:::fail n2["Crates/orbital-iced/futures/src/subscription.rs - subscription::Subscr"]:::fail n0 --> n1 --> n2 classDef fail fill:#d73a49,color:#ffffff,stroke:#ffffff ``` <details> <summary>Failure details</summary> #### iced_futures::Crates/orbital-iced/futures/src/subscription.rs - subscription::Subscription (line 94) <pre><code>error[E0432]: unresolved import `iced_futures::backend::default::time::every` --&gt; Crates/orbital-iced/futures/src/subscription.rs:98:17 | 98 | pub use iced_futures::backend::default::time::every; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^----- | | | no `every` in `backend::native::thread_pool::time` error: aborting due to 1 previous error For more information about this error, try `rustc --explain E0432`. Couldn&#39;t compile the test.</code></pre> </details>
ci: polish smart test reporting
Some checks failed
Lint / lint (pull_request) Successful in 1m22s
PR / test (pull_request) Failing after 20m8s
0f686f25c1
ci: split nextest and doctest outcome reporting
Some checks failed
Lint / lint (pull_request) Successful in 1m25s
PR / test (pull_request) Failing after 20m52s
34d3b078a7
Sign in to join this conversation.
No description provided.