fix(e2e): remove epoch gap flakiness by jpraynaud · Pull Request #3445 · IntersectMBO/mithril

jpraynaud · 2026-07-30T17:09:38Z

Content

This PR includes the removal of several sources of flakiness in the e2e tests which caused intermittent Timeout exhausted waiting for Certificate for epoch XX failures in the CI:

Poll epoch transitions at a tenth of the epoch duration, so that target epochs are detected shortly after their boundary instead of up to a full epoch late with the previous exponential backoff
Certify the current epoch before the protocol parameters restart of the aggregator, so that the restart can no longer create an unrecoverable epoch gap
Extend the artifact production timeout from one to three epochs: an artifact type is produced once per epoch, so a wait started late in an epoch may need to span a full production cycle before the awaited artifact exists, which a single epoch timeout could not cover
Lighten the updated protocol parameters by reducing the number of lotteries played, closer to the initial parameter set, as the test only needs to exercise the parameters update

The fixed e2e test has been run 381 times with 1 failure, a 99.74% success ratio.

Pre-submit checklist

Branch
- Crates versions are updated (if relevant)
- Commit sequence broadly makes sense
- Key commits have useful messages
PR
- All check jobs of the CI have succeeded
- Self-reviewed the diff
- Useful pull request description
- Reviewer requested

Issue(s)

Closes #3452

github-actions · 2026-07-30T17:24:44Z

Test Results

5 files ±0 209 suites ±0 57m 19s ⏱️ - 2m 11s
3 331 tests +1 3 331 ✅ +1 0 💤 ±0 0 ❌ ±0
11 093 runs +3 11 093 ✅ +3 0 💤 ±0 0 ❌ ±0

Results for commit 520f12b. ± Comparison against base commit 71ea932.

♻️ This comment has been updated with latest results.

The exponential poll backoff exceeded short devnet epochs, so target epochs could be detected up to a full epoch after their boundary.

A restart crossing an epoch boundary before the current epoch is certified creates an unrecoverable epoch gap that blocks the aggregator.

A single epoch left no slack for certificate sealing latency, causing spurious 'Timeout exhausted waiting for...' failures.

Reduce the number of lotteries played to get closer to the initial parameter set, as the test only needs to exercise the parameters update.

* mithril-end-to-end from `0.5.9` to `0.5.10`

Copilot

Pull request overview

This PR reduces intermittent e2e CI failures caused by epoch-boundary timing and aggregator restarts by tightening epoch transition polling, ensuring certificate continuity across restarts, and increasing artifact wait timeouts.

Changes:

Switch epoch-transition waiting to a constant “tenth of epoch” polling cadence.
Ensure the current epoch is certified before restarting the aggregator for protocol-parameter updates.
Increase artifact production timeout to cover up to three epochs and reduce concatenation protocol parameters used during the update.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
mithril-test-lab/mithril-end-to-end/src/toolkit/wait.rs	Uses the new constant tenth-of-epoch backoff when waiting for target epoch transitions.
mithril-test-lab/mithril-end-to-end/src/toolkit/exec.rs	Lightens concatenation protocol parameters used during the update/restart step.
mithril-test-lab/mithril-end-to-end/src/toolkit/context.rs	Adds `tenth_of_epoch_poll_backoff()` and increases artifact production timeout to 3 epochs (plus tests).
mithril-test-lab/mithril-end-to-end/src/scenario/full.rs	Adds a pre-restart certification check to prevent unrecoverable epoch gaps.
mithril-test-lab/mithril-end-to-end/Cargo.toml	Bumps `mithril-end-to-end` crate version to `0.5.10`.
Cargo.lock	Updates lockfile for the `mithril-end-to-end` version bump.

+            // A certificate chain lagging one epoch behind the chain tip is normal, but a restart
+            // crossing an epoch boundary before the current epoch is certified would create an
+            // unrecoverable epoch gap and block the aggregator
+            let current_epoch = chain_observer.get_current_epoch().await?.unwrap_or_default();


Alenar · 2026-08-04T09:12:27Z

+        let delay = self.timeout_for_epochs(1) / 10;
+        Backoff::new(delay, delay, 1)


This is true: https://play.rust-lang.org/?version=stable&mode=release&edition=2024&gist=c1895e5a63c9bf02f672eb5b6cea6dd1

Maybe we should add a timeout_for_epoch_fraction(&self, fraction: u32) -> Duration to centralize the behavior?

turmelclem

LGTM 👍

Alenar

LGTM

Alenar · 2026-08-04T08:42:28Z

    /// Timeout to wait for the aggregator to produce a signed artifact once it is running.
    pub fn artifact_production_timeout(&self) -> Duration {
-        self.timeout_for_epochs(1)
+        self.timeout_for_epochs(3)


This timeout is also used for is_creating_certificate_with_enough_signers which does not make sense with a timeout of three epochs has the certificate hash is provided so it already exist (the timeout here only make sense to handle http request issues).
So I would change is_creating_certificate_with_enough_signers timeout to use timeout_for_epochs(1) or better a new named timeout method.

Alenar · 2026-08-04T09:01:54Z

+            // A certificate chain lagging one epoch behind the chain tip is normal, but a restart
+            // crossing an epoch boundary before the current epoch is certified would create an
+            // unrecoverable epoch gap and block the aggregator


It took me some time to understand the problem solved here, I think that sightly changing the text would make it more clear (with a first line saying the goal achieve, and the rest why we need it)

Suggested change

// A certificate chain lagging one epoch behind the chain tip is normal, but a restart

// crossing an epoch boundary before the current epoch is certified would create an

// unrecoverable epoch gap and block the aggregator

// Ensure the current epoch is certified

// Given the time needed to restart the aggregator, a restart crossing an epoch

// boundary before the current epoch is certified would create an

// unrecoverable epoch gap and block the aggregator

Alenar · 2026-08-04T09:12:27Z

+        let delay = self.timeout_for_epochs(1) / 10;
+        Backoff::new(delay, delay, 1)


This is true: https://play.rust-lang.org/?version=stable&mode=release&edition=2024&gist=c1895e5a63c9bf02f672eb5b6cea6dd1

Maybe we should add a timeout_for_epoch_fraction(&self, fraction: u32) -> Duration to centralize the behavior?

jpraynaud self-assigned this Jul 30, 2026

jpraynaud temporarily deployed to testing-preview July 30, 2026 18:00 — with GitHub Actions Inactive

jpraynaud temporarily deployed to testing-2-preview July 30, 2026 18:00 — with GitHub Actions Inactive

jpraynaud temporarily deployed to testing-preview July 30, 2026 19:23 — with GitHub Actions Inactive

jpraynaud temporarily deployed to testing-2-preview July 30, 2026 19:23 — with GitHub Actions Inactive

jpraynaud temporarily deployed to testing-preview July 30, 2026 20:45 — with GitHub Actions Inactive

jpraynaud temporarily deployed to testing-2-preview July 30, 2026 20:45 — with GitHub Actions Inactive

jpraynaud temporarily deployed to testing-2-preview July 31, 2026 07:58 — with GitHub Actions Inactive

jpraynaud temporarily deployed to testing-preview July 31, 2026 07:58 — with GitHub Actions Inactive

jpraynaud force-pushed the jpraynaud/fix-e2e-flakiness branch 4 times, most recently from 4847826 to 4f65f57 Compare August 3, 2026 06:30

jpraynaud temporarily deployed to testing-preview August 3, 2026 06:48 — with GitHub Actions Inactive

jpraynaud temporarily deployed to testing-2-preview August 3, 2026 06:48 — with GitHub Actions Inactive

jpraynaud temporarily deployed to testing-preview August 3, 2026 10:43 — with GitHub Actions Inactive

jpraynaud temporarily deployed to testing-2-preview August 3, 2026 10:43 — with GitHub Actions Inactive

jpraynaud force-pushed the jpraynaud/fix-e2e-flakiness branch from 4f65f57 to 8ad0e8c Compare August 3, 2026 14:59

jpraynaud added 5 commits August 3, 2026 17:38

fix(e2e): poll epoch transitions at a tenth of the epoch duration

92331fe

The exponential poll backoff exceeded short devnet epochs, so target epochs could be detected up to a full epoch after their boundary.

fix(e2e): certify current epoch before protocol parameters restart

92d5641

A restart crossing an epoch boundary before the current epoch is certified creates an unrecoverable epoch gap that blocks the aggregator.

fix(e2e): extend artifact production timeout to three epochs

9bb7e7d

A single epoch left no slack for certificate sealing latency, causing spurious 'Timeout exhausted waiting for...' failures.

fix(e2e): lighten updated protocol parameters

31508be

Reduce the number of lotteries played to get closer to the initial parameter set, as the test only needs to exercise the parameters update.

chore: upgrade crate versions

520f12b

* mithril-end-to-end from `0.5.9` to `0.5.10`

jpraynaud force-pushed the jpraynaud/fix-e2e-flakiness branch from 8ad0e8c to 520f12b Compare August 3, 2026 15:51

jpraynaud marked this pull request as ready for review August 3, 2026 15:52

jpraynaud requested a review from Alenar as a code owner August 3, 2026 15:52

Copilot AI review requested due to automatic review settings August 3, 2026 15:52

jpraynaud requested a review from turmelclem as a code owner August 3, 2026 15:52

Copilot started reviewing on behalf of jpraynaud August 3, 2026 15:53 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

jpraynaud temporarily deployed to testing-2-preview August 3, 2026 16:09 — with GitHub Actions Inactive

jpraynaud temporarily deployed to testing-preview August 3, 2026 16:09 — with GitHub Actions Inactive

turmelclem approved these changes Aug 4, 2026

View reviewed changes

Alenar approved these changes Aug 4, 2026

View reviewed changes

Navigation Menu

Uh oh!

fix(e2e): remove epoch gap flakiness - #3445

fix(e2e): remove epoch gap flakiness#3445
jpraynaud wants to merge 5 commits into
mainfrom
jpraynaud/fix-e2e-flakiness

jpraynaud commented Jul 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Alenar Aug 4, 2026 •

edited

Loading

Uh oh!

turmelclem left a comment

Uh oh!

Alenar left a comment

Uh oh!

Alenar Aug 4, 2026

Uh oh!

Alenar Aug 4, 2026

Uh oh!

Alenar Aug 4, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		let delay = self.timeout_for_epochs(1) / 10;
		Backoff::new(delay, delay, 1)

Uh oh!

Conversation

jpraynaud commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Content

Pre-submit checklist

Issue(s)

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Alenar Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

turmelclem left a comment

Choose a reason for hiding this comment

Uh oh!

Alenar left a comment

Choose a reason for hiding this comment

Uh oh!

Alenar Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Alenar Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Alenar Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jpraynaud commented Jul 30, 2026 •

edited

Loading

github-actions Bot commented Jul 30, 2026 •

edited

Loading

Alenar Aug 4, 2026 •

edited

Loading

Alenar Aug 4, 2026 •

edited

Loading