Skip to content

ci: skip the device test app in the macOS build and stop serializing verify-api and format-code - #5758

Merged
jamescrosswell merged 2 commits into
mainfrom
ci/faster-pr-builds-round-2
Oct 22, 2026
Merged

jamescrosswell merged 2 commits into
mainfrom
ci/faster-pr-builds-round-2

Conversation

@ric-oliv

@ric-oliv ric-oliv commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

This follows up on #5746. On #5752, the first version7 PR after it, the build workflow took 63.3 minutes, and the macOS .NET job took 59.6 of those. This PR makes two more cuts, and no test is dropped. The device tests now also run for a few changes they used to skip.

I first opened it against version7 and measured it there, on top of #5746. Neither change depends on #5746, so it now targets main.

  • Sentry-CI-Build-macOS.slnf no longer includes Sentry.Maui.Device.TestApp. The macOS build job compiled it for net9.0-android, net10.0-android and net10.0-ios (net10.0-android, net11.0-android and net11.0-ios on version7), and nothing in build.yml runs it. In chore(samples): move the Azure Functions sample to Azure.Functions.Sdk and net11.0 #5752's run on version7, packaging that app was all the Build step did after minute 18.4 of 24.7, and in a recent main run it took the last 8 minutes of 35. The Linux and Windows filters already exclude it, and both device test workflows build it themselves. On main, the macOS leg of verify-api builds the same filter, so it stops building the app too.
    • Nothing in the build workflow ever used what it built. dotnet test skips it because platform TFMs aren't test projects here, dotnet pack skips it because it isn't packable, and the sparse checkout deletes its bin and obj before the integration tests. So all it gave us was a compile check, and both device test workflows already run one with scripts/device-test.ps1 -Build. Every project it depends on still builds for its mobile TFMs, and the job packs the same packages as before.
    • To keep that safe, the device test path filters now also list what the app compiles and they didn't: Sentry.Analyzers, Sentry.Testing.CrashableApp, the nested Directory.Build.* files and .editorconfig, which can fail the build because warnings are errors.
    • On version7, both filters also need src/Sentry.Extensions.Hosting/**, since feat: add Sentry.Extensions.Hosting with UseSentry for generic host apps #5739 made Sentry.Maui.Tests reference it. That project isn't on main, so I left it out here. It needs adding when main next merges into version7.
  • verify-api and format-code shared the pr-auto-commit concurrency group at workflow level, so each waited for the other to finish. On one PR, format-code started 35 minutes after its run was created, and on another verify-api waited 13 minutes. The group exists to keep their git pushes apart, so it now covers only accept-api-changes and the format-code job.

Here's the CI run on main, compared with the medians of main PR runs since October 15, which are 26 build runs and 14 verify-api runs:

main PRs, median This PR on main
Build workflow 78.9 min 66.7 min
macOS .NET job 67.0 min 53.2 min
macOS Build step 32.7 min 22.9 min
.NET (win-x64) job 51.9 min 55.8 min
verify-api macOS job 35.7 min 29.6 min
  • That's about 12 minutes off the build workflow. It would have been about 8 more, but Unsupported AOT (win-x86), which waits on every leg, spent 8 minutes in setup on this run.
  • The macOS job ran the same 59 test runs with the same 11,451 passing tests as a recent main PR and packed the same 53 packages. format-code was done after 12.5 minutes instead of waiting for the API tests, and the device tests passed.

Here's what the build workflow took on three CI runs of this PR while it targeted version7, compared with #5752:

#5752 Run 1 Run 2 Run 3
Build workflow 63.3 min 62.9 min 56.4 min 56.7 min
macOS .NET job 59.6 min 57.8 min 48.1 min 49.0 min
macOS Build step 24.7 min 19.8 min 15.6 min 15.7 min
.NET (win-x64) job 48.8 min 53.4 min 53.3 min 53.9 min

So on version7 this PR saves about 7 minutes on every build workflow run. Runs 2 and 3 took 56.4 and 56.7 minutes, 6.6 to 6.9 minutes under #5752. Run 1 barely moved because that runner spent 4.5 minutes downloading one workload pack during setup and ran the tests 2.4 minutes slower.

With macOS down to about 49 minutes on version7 and 53 on main, .NET (win-x64) is now the slowest leg on both, at 53 to 56 minutes. So the next round of CI work should focus on the Windows jobs. Their Test step alone takes 16 to 21 minutes, and in #5752's run only about 8 of those were spent inside the tests themselves.

The first two runs also included an iOS change that I've since taken out: booting the simulator while the device test app builds. It cut the app install from 6 min 46 s to under 20 seconds, but the app build got slower by about the same amount, so the iOS job didn't get faster. The build workflow doesn't use either of the files it touched, and run 3 ran without it.

Craft releases aren't affected. The device tests still run on every push to main and release/*, the macOS job still uploads the commit-SHA artifact with the same packages, and no check names change.

#skip-changelog

@codecov

codecov Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.13%. Comparing base (04cac0d) to head (aa12ec7).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5758   +/-   ##
=======================================
  Coverage   76.13%   76.13%           
=======================================
  Files         528      528           
  Lines       19579    19579           
  Branches     3829     3829           
=======================================
  Hits        14906    14906           
  Misses       3791     3791           
  Partials      882      882           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ric-oliv added a commit that referenced this pull request Oct 21, 2026
device-test.ps1 picked a simulator in -Build too, though only xharness uses it. Now that the workflow boots the simulator while the app builds, that simctl call blocked for about 3 minutes on #5758's first run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ric-oliv
ric-oliv force-pushed the ci/faster-pr-builds-round-2 branch from a6c8173 to 6ddf5e1 Compare October 21, 2026 18:58
@ric-oliv ric-oliv changed the title ci: skip the device test app in the macOS build and boot the iOS simulator early ci: skip the device test app in the macOS build and stop serializing verify-api and format-code Oct 21, 2026
@ric-oliv
ric-oliv marked this pull request as ready for review October 21, 2026 20:11
@github-actions github-actions Bot added the risk: medium PR risk score: medium label Oct 21, 2026

@jamescrosswell jamescrosswell left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice - thanks @ric-oliv !

Any specific reason to target version7 with this change? If not, maybe we could change the target to main before merging?

ric-oliv and others added 2 commits October 22, 2026 09:40
The macOS leg of build.yml built Sentry.Maui.Device.TestApp for net9.0-android, net10.0-android
and net10.0-ios, and nothing in build.yml runs it. In the run for #5752 on version7 that was the
last 6 minutes of the 24.7-minute Build step, on the critical path. The Linux and Windows filters
already exclude the app, and both device test workflows build it themselves. The macOS leg of
verify-api builds the same filter, so it no longer builds the app either.

The device test path filters now also cover what the app compiles and they didn't list:
Sentry.Analyzers, Sentry.Testing.CrashableApp, the nested Directory.Build files and .editorconfig,
which can fail the build because warnings are errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both workflows put the whole run in the pr-auto-commit concurrency group, so each waited for the
other to finish. On one PR, format-code started 35 minutes after its run was created, and on
another verify-api waited 13 minutes. The group exists to keep their git pushes apart, so it now
covers only accept-api-changes and the format-code job, and the API test matrix starts right
away. Both still push from the head they checked out and fail as non-fast-forward if the branch
moved in the meantime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ric-oliv
ric-oliv changed the base branch from version7 to main October 22, 2026 08:06
@ric-oliv
ric-oliv force-pushed the ci/faster-pr-builds-round-2 branch from 6ddf5e1 to aa12ec7 Compare October 22, 2026 08:06
@ric-oliv

Copy link
Copy Markdown
Member Author

Good point, no reason. Changed to main!

@jamescrosswell jamescrosswell left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice - thanks @ric-oliv !

@jamescrosswell
jamescrosswell merged commit 418367a into main Oct 22, 2026
50 checks passed
@jamescrosswell
jamescrosswell deleted the ci/faster-pr-builds-round-2 branch October 22, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk: medium PR risk score: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants