Skip to content

Fix BSL count race condition in multi-BSL e2e test - #2498

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift:oadp-1.4from
redhat-chai-bot:fix/bsl-count-race-oadp-1.4
Oct 20, 2026
Merged

openshift-merge-bot[bot] merged 1 commit into
openshift:oadp-1.4from
redhat-chai-bot:fix/bsl-count-race-oadp-1.4

Conversation

@redhat-chai-bot

Copy link
Copy Markdown
Contributor

Problem

The periodic job periodic-ci-openshift-oadp-operator-oadp-1.4-4.18-e2e-test-aws-periodic has been failing with 2 consecutive failures (Oct 18–19) due to a race condition in the multi-BSL e2e test.

In tests/e2e/backup_restore_suite_test.go at line 673, the BSL count assertion uses a synchronous gomega.Expect immediately after a DPA patch. The operator may not have finished reconciling, so ListBSLs() can return a stale count (fewer than 3 BSLs), causing the test to fail.

Root Cause

IsReconciledTrue() does not compare generation vs observedGeneration, so it can return True from a prior reconciliation cycle. The test then proceeds to check the BSL count before all 3 BSLs have been created by the new reconciliation.

Fix

Replace the synchronous assertion:

gomega.Expect(len(bsls.Items)).To(gomega.Equal(3), "Should have 3 BSLs configured")

With an asynchronous poll using gomega.Eventually:

gomega.Eventually(func() int {
    bsls, err := dpaCR.ListBSLs()
    if err != nil {
        return -1
    }
    return len(bsls.Items)
}, time.Minute*3, time.Second*5).Should(gomega.Equal(3), "Should have 3 BSLs configured")

The bsls variable is re-assigned after the Eventually succeeds so that downstream code (finding the default BSL) continues to work.

Validation

  • go vet ./tests/e2e/... — passes
  • go build ./tests/e2e/... — passes

Failing job run

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-oadp-operator-oadp-1.4-4.18-e2e-test-aws-periodic/2112031248795832320


AI-generated. Review for accuracy.

@kaovilai requested from Slack

The BSL count assertion at line 673 uses a synchronous gomega.Expect
that can fail when the operator has not finished reconciling after a
DPA patch. Replace with gomega.Eventually to poll until all 3 BSLs
are present, with a 3-minute timeout and 5-second polling interval.

This fixes flaky failures in periodic-ci-openshift-oadp-operator-oadp-1.4-4.18-e2e-test-aws-periodic.
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification

This PR uses the pipeline controller for second-stage tests. Selection and triggering follow the repository configuration.

Use /test ? to list jobs and /pipeline help for commands.

@openshift-ci
openshift-ci Bot requested review from kaovilai and weshayutin October 19, 2026 16:47
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 40.03%. Comparing base (6afedcf) to head (8761f21).
⚠️ Report is 2 commits behind head on oadp-1.4.

⚠️ Current head 8761f21 differs from pull request most recent head 9d0f6ad

Please upload reports for the commit 9d0f6ad to get more accurate results.

Additional details and impacted files
@@            Coverage Diff            @@
##           oadp-1.4    #2498   +/-   ##
=========================================
  Coverage     40.03%   40.03%           
=========================================
  Files            30       30           
  Lines          5378     5378           
=========================================
  Hits           2153     2153           
  Misses         3014     3014           
  Partials        211      211           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai

coderabbitai Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 8a8f0454-b01d-48f8-ae1f-6b5caf7e2c62

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

/retest

@kaovilai kaovilai left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@openshift-ci openshift-ci Bot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Oct 19, 2026
@openshift-ci

openshift-ci Bot commented Oct 19, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kaovilai, redhat-chai-bot, shubham-pampattiwar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

/retest

@openshift-ci

openshift-ci Bot commented Oct 20, 2026

Copy link
Copy Markdown

@redhat-chai-bot: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
openshift-merge-bot Bot merged commit 75dc5ec into openshift:oadp-1.4 Oct 20, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants