Skip to content

Keep tests with identical names in different files separate - #668

Open
andrcuns wants to merge 1 commit into
masterfrom
fix-tc-merge
Open

andrcuns wants to merge 1 commit into
masterfrom
fix-tc-merge

Conversation

@andrcuns

@andrcuns andrcuns commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

Context

Cucumber scenarios with the same name in different feature files, or RSpec examples with the same full description in different spec files, can be merged into a single test in Allure. This can hide a real failure as a retry when another test with the same name passes.

This change gives tests stable, file-aware identities so that tests in different files remain separate, even when their displayed names match. These identities do not depend on source line numbers or example order. Cucumber scenario outline rows share the outline's identity, with their parameters distinguishing the individual variants.

Test history compatibility: This change may affect test history in the report. After upgrading, some tests may appear as new, and their history may restart or no longer link to results generated with earlier adapter versions because Allure now identifies them differently.

Checklist

@andrcuns andrcuns added the pr:improvement Feature or code improvements label Nov 3, 2026
@andrcuns andrcuns self-assigned this Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

Allure Report Summary

     Environment Duration Stats New Flaky Retry
     All tests 4s Passed tests⁠ 1212 1212 0 0
     ubuntu-ruby-3_3-oj-false 699ms Passed tests⁠ 202 202 0 0
     ubuntu-ruby-3_3-oj-true 544ms Passed tests⁠ 202 202 0 0
     ubuntu-ruby-3_4-oj-false 773ms Passed tests⁠ 202 202 0 0
     ubuntu-ruby-3_4-oj-true 457ms Passed tests⁠ 202 202 0 0
     ubuntu-ruby-4_0-oj-false 525ms Passed tests⁠ 202 202 0 0
     ubuntu-ruby-4_0-oj-true 800ms Passed tests⁠ 202 202 0 0

Allure TestOps: Allure Ruby GitHub actions run (2026-11-03T11:51:59.742Z)

Artifacts used (6)
  • allure-results-test-ruby-3.3-oj-false.zip
  • allure-results-test-ruby-3.3-oj-true.zip
  • allure-results-test-ruby-3.4-oj-false.zip
  • allure-results-test-ruby-3.4-oj-true.zip
  • allure-results-test-ruby-4.0-oj-false.zip
  • allure-results-test-ruby-4.0-oj-true.zip

@andrcuns
andrcuns marked this pull request as ready for review November 3, 2026 16:27
@andrcuns andrcuns added pr:breaking change Change introducing a breaking behavior pr:bug and removed pr:improvement Feature or code improvements labels Nov 3, 2026
@andrcuns
andrcuns requested a review from epszaw November 3, 2026 16:28
@andrcuns andrcuns removed the pr:bug label Nov 3, 2026
@andrcuns

andrcuns commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

@epszaw One thing I'm noticing here in this PR, why do we show all tests as "new"? Is it because the summary table is always generated from the report, not testops, right?

If so, I think we should definitely look in to this, it will be incredibly confusing for anyone using this summary table in combination with testops. Completely unrelated to this change, just mentioning it before I forget 😄

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr:breaking change Change introducing a breaking behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant