Skip to content

Make failure dumps downloadable from CI - #1077

Merged
maleadt merged 1 commit into
mainfrom
tb/failure-dumps
Nov 5, 2026
Merged

maleadt merged 1 commit into
mainfrom
tb/failure-dumps

Conversation

@maleadt

@maleadt maleadt commented Nov 5, 2026

Copy link
Copy Markdown
Member

When a back-end compilation or a SPIR-V validation/optimization step fails, GPUCompiler writes the module to a temporary file and asks the user to attach it to a bug report. On CI that file disappears with the runner. That's what happened with a one-off Failed to validate generated SPIR-V: ID '297[%297]' has not been defined in AcceleratedKernels.jl's CI. Locally, the SPIR-V back-end produces identical, valid output for every module of that test, so only the dumped bitcode could explain the failure.

This routes those dumps through a single dump_failure helper:

  • on GitHub Actions, the files go to $RUNNER_TEMP/gpucompiler-dumps, with a ::notice, so that an actions/upload-artifact step can collect them;
  • on Buildkite, they are uploaded with buildkite-agent artifact upload;
  • elsewhere, they go to tempdir() as before.

The .bc and .spv files of a failed SPIR-V step now share a base name. The mechanism follows the one from 0d5bdc4, which #861 removed because it serialized every module before optimization. This version only runs once a compilation has failed, so it costs nothing otherwise.

When a back-end or SPIR-V validation/optimization fails, the module is written to a
temporary file to attach to a bug report. On CI that file disappears with the runner,
so a failure that doesn't reproduce locally can't be debugged.

Write these dumps through one helper that, on GitHub Actions, puts them in
`$RUNNER_TEMP/gpucompiler-dumps` (with a notice) for an `actions/upload-artifact` step
to collect, and on Buildkite uploads them as artifacts. The input and output of a
failed SPIR-V step now share a base name. Unlike the dump removed in #861, this only
runs once a compilation has failed, so it costs nothing otherwise.
@maleadt
maleadt merged commit eec1950 into main Nov 5, 2026
32 of 36 checks passed
@maleadt
maleadt deleted the tb/failure-dumps branch November 5, 2026 13:25
@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 76.19048% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.74%. Comparing base (709c4cc) to head (3cac374).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
src/utils.jl 75.00% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1077      +/-   ##
==========================================
+ Coverage   92.65%   92.74%   +0.09%     
==========================================
  Files          31       31              
  Lines        7501     7514      +13     
==========================================
+ Hits         6950     6969      +19     
+ Misses        551      545       -6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant