Skip to content

ci: run the Android integration tests in their own job - #5800

Draft
ric-oliv wants to merge 1 commit into
mainfrom
ci/split-android-integration-tests
Draft

ric-oliv wants to merge 1 commit into
mainfrom
ci/split-android-integration-tests

Conversation

@ric-oliv

@ric-oliv ric-oliv commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

Each Android emulator job runs the device tests and then the integration tests. The integration tests don't use anything from the device tests. They build their own app, Sentry.Maui.Device.IntegrationTestApp, from source, and only share the emulator.

Here's the longest emulator job from #5795's run, API-34 on net10.0, with Setup Environment at its usual 3.1 minutes because #5794 removes the Mono download:

Step Minutes
Wait for both device test app builds 7.5
Device tests, with the emulator boot 3.2
Setup Environment, only needed by the integration tests 3.1
Integration tests 14.9

9.9 of those 14.9 minutes are the 4 integration app builds, while the emulator sits idle.

So this PR moves android.Tests.ps1 into a second job, android-integration-tests, with the same TFM and API-level matrix. It doesn't need the device test app, so it starts with the workflow.

The new job's first emulator boot also installs the system image, which took 1.6 to 2.8 minutes in #5795's run, against about 0.6 for the second boot today. With that, I expect:

Minutes Today, with #5794 This PR
Wait for the device test app builds 7.5 7.5
Emulator job with the device tests 21.9, with the integration tests 3 to 5
Emulator job with the integration tests about 20, from the start
Android Device Tests about 29 about 20

With the other open CI PRs, the build workflow gets to about 30 minutes, and Android at about 29 would be right behind it. This should bring Android to about 20, and an integration test retry, 6 to 15 minutes, no longer stacks on the device tests.

Measured on the first run, against #5795's run from the same evening, which had no Mono download either:

Without this PR This PR
Android Device Tests 30.1 min, and 28.9 on #5792 19.8
Emulator jobs with the device tests 11.7 to 22.5 3.6 to 3.8
Emulator jobs with the integration tests 12.0 to 19.7, from the start
Run Integration Tests step 5.9 to 15.2 8.4 to 15.9

All 8 jobs passed. The device test jobs now take under 4 minutes after the test app builds, and the longest integration job, API-36 on net10.0, sets the workflow's time.

This and #5795 both edit the end of the android job, so whichever lands second needs a small rebase around #5795's last "Show disk space" step.

#skip-changelog

Each Android emulator job ran the device tests, then Setup Environment,
then the integration tests, about 22 minutes after waiting 7.5 for the
device test app builds. The integration tests build their own app from
source and share only the emulator, so they now run in a second job that
starts with the workflow.

The new job drops "Select Java 21". It came with #4559, when Setup
Environment ended on Java 11; since #5060 it installs only Java 21.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.12%. Comparing base (b41b96c) to head (f6b4b03).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5800      +/-   ##
==========================================
- Coverage   76.13%   76.12%   -0.01%     
==========================================
  Files         528      528              
  Lines       19579    19579              
  Branches     3829     3829              
==========================================
- Hits        14906    14905       -1     
  Misses       3791     3791              
- Partials      882      883       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant