Flaky tests: TestReactorValidatorSetChanges and TestRouter_EvictPeers time out intermittently on v1.6-dev

[lklimek]

Summary

Two timing-sensitive tests fail intermittently on v1.6-dev itself, unrelated to any PR under test. They currently cost dependency PRs full CI re-runs and make green/red an unreliable merge signal.

Affected tests

Test	Package
TestReactorValidatorSetChanges	internal/consensus
TestRouter_EvictPeers	internal/p2p

Evidence — both reproduce on the base branch

These are not PR-induced. Observed on v1.6-dev with no feature branch involved:

Run	Commit	Failing test
30377407879	2e125053	TestReactorValidatorSetChanges
30381042778	c4063ef0	TestRouter_EvictPeers

The same TestReactorValidatorSetChanges failure also appeared on #1397 — a go.mod-only Dependabot bump that cannot reach consensus code at all (buf sits behind //go:build tools; go list -deps ./... resolves zero bufbuild packages). Re-running the job passed.

Failure shape

Both are timeouts rather than assertion failures. TestReactorValidatorSetChanges:

reactor_test.go:192: waitForAndValidateBlock deadline for node 6: context deadline exceeded
reactor_test.go:192: waitForAndValidateBlock deadline for node 0: context deadline exceeded
reactor_test.go:628: encountered timeout: context deadline exceeded
FAIL github.com/dashpay/tenderdash/internal/consensus  167.820s

The trailing sync: autofile is closed WAL errors are teardown noise after the deadline fires, not the cause.

The test spins up a 7-validator network and waits for block production within a fixed deadline; on a loaded or slow CI runner the network doesn't converge in time. 167s runtime suggests it is running close to its budget even when it passes.

Impact

• Merge decisions on unrelated PRs need manual triage to distinguish flake from regression.
• Every occurrence costs a CI re-run — significant while the dependency backlog is being processed.
• Erodes trust in a red build, which is the more expensive long-term cost.

Possible directions

1. Instrument first — record pass/fail and duration to see how close to the deadline passing runs sit; the fix differs depending on whether this is a marginal timeout or a genuine liveness bug.
2. If marginal: raise the deadline, or scale it from a -timeout-derived budget rather than a hard-coded constant.
3. If it is a real convergence stall, the timeout is masking a consensus liveness bug and should be investigated rather than lengthened.
4. Consider -race-off or reduced parallelism for these packages if runner contention is the trigger.

Worth checking whether these two share a root cause (runner contention) or are independent.

Context

Surfaced while auditing the Dependabot backlog — see #1397, where it caused a false red on a go.mod-only change.

_{🤖 Reported by Claudius the Magnificent AI Agent}