Skip to content

ci: measure the round 4 and 5 CI changes together (do not merge) - #5793

Closed
ric-oliv wants to merge 9 commits into
mainfrom
ci/measure-combined-round4-changes
Closed

ric-oliv wants to merge 9 commits into
mainfrom
ci/measure-combined-round4-changes

Conversation

@ric-oliv

Copy link
Copy Markdown
Member

This draft is only here to measure CI with the round 4 and round 5 changes applied together. Please don't review or merge it. I'll close it once the runs finish.

It's the branch from #5788, so main at the time plus these PRs, applied in order:

#5788 measured the first four at 35.2 minutes for the build workflow, while the iOS job took about 42. This run checks how close the slowest check on a PR gets to the build workflow once the iOS job is split.

The only conflict was in the build legs, where #5785 and #5791 both add steps after the integration test. The unsupported AOT test runs before the wait for the background tests, so the two overlap.

#skip-changelog

ric-oliv and others added 9 commits October 23, 2026 23:01
The app samples only check that they compile, but their Release builds trim,
AOT-compile and package each app. In the macOS leg that's about 13.7 of the
Build step's 28.3 task-minutes, and it's the whole tail of the step once the
Cocoa SDK comes from the cache. On Windows, the MAUI sample's Android build
is the last thing the Build step finishes.

The Android, iOS, Mac Catalyst, macOS and MAUI samples now build in a
separate job that starts with the .NET legs. It builds all of them on macOS,
and the MAUI sample on Windows, which is the only build of its Windows
target. The legs' solution filters no longer include them, and
Sentry-CI-Build-Samples-macOS.slnf lists them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Windows .NET legs spend about 12 minutes in Setup Environment and
Restore, against about 4.5 on Linux. On the Windows images, setup-dotnet
installs into C:\Program Files\dotnet, where the SDK installs workloads as
MSI packages, and NuGet extracts packages into the user profile on C:. The
workload step takes 5.6 minutes there, against about 2 on macOS and Linux,
which install more workloads.

This installs .NET and the NuGet cache next to the workspace, where
workloads install as plain files. On the x64 runners that also moves them to
D:, and on the arm64 runners they stay on C:, so the two legs separate the
install mode from the drive. Windows also stops restoring the Android test
app, which its solution filters exclude.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Setup Environment runs dotnet workload restore without a project, so it
evaluates every project in Sentry.slnx on every leg. The three Linux legs
without mobile targets install android, maui-android, maui-tizen, macos,
wasm-tools and wasm-tools-net9. Windows installs ios, maccatalyst and macos.
Every leg installs maui-tizen, which nothing targets, and macos, which only
the macOS sample needs.

The build legs now pass their solution filter, and the step restores the
workloads for the projects in it. dotnet workload restore doesn't accept a
solution filter, so the step passes its projects. The Android test app is
restored only when the filter includes the tests that build it, and the legs
without mobile targets skip Java and the Android SDK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the build legs, the unit tests and the integration tests run one after
the other, though the integration tests only need the packages. On win-x64
the Test step takes 5.8 minutes and Integration test 14.

Outside the publishing build, Test now runs as a background step. The job
packs and runs the integration tests next to it, then waits for it before
uploading coverage. The macOS publishing build keeps the current order,
because its sparse checkout removes the test outputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every solution build runs before.Sentry.sln.targets, which calls
git submodule update --init --recursive. With the tests in the background,
dotnet test and dotnet pack started in the same second on linux-x64, and
the second git call failed on the lock for a submodule's config file, so
the tests failed before they ran. Packing takes under a minute, so the
tests now start once it's done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The iOS job ran the device tests and then the integration tests on one
Mac, about 42 minutes in all. The two build and run different apps and
share only the simulator, so the integration tests now run in a second
job at the same time.

ios.Tests.ps1 skips every case unless a simulator is already booted, and
xharness only boots one in the ios-tests job, so the new job boots its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Build step wrote a detailed text log next to the binlog, 1.9 GB on
macOS, and uploaded both. The binlog holds the same events and warnings,
and nothing reads the text log. The MSBuild job and the integration test
builds wrote one too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Unsupported AOT jobs waited for the whole build matrix to use the
macOS packages, but any leg's packages work. Every leg already packs and
runs aot.Tests.ps1 against its own. linux-arm now runs in the linux-arm64
leg and win-x86 in the win-x64 leg, which removes two jobs and the tail
after the matrix.

Some arm64 hosts can't run 32-bit ARM binaries and failed with "Exec
format error", so QEMU runs them on every host. The leg also logs whether
its host runs them natively.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A registered QEMU handler doesn't prove it ran the binary. A 32-bit ARM
process sees armv8l when the arm64 kernel runs it and armv7l under QEMU,
so the leg compiles a tiny uname program, logs what the host does
natively, and fails unless QEMU runs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.14%. Comparing base (ff0355e) to head (5c5733b).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5793      +/-   ##
==========================================
+ Coverage   76.13%   76.14%   +0.01%     
==========================================
  Files         528      528              
  Lines       19579    19579              
  Branches     3829     3829              
==========================================
+ Hits        14906    14909       +3     
+ Misses       3791     3788       -3     
  Partials      882      882              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant