Skip to content

Gate FA4 and stabilize attention test imports - #3341

Open
sudhakarsingh27 wants to merge 1 commit into
NVIDIA:mainfrom
sudhakarsingh27:sudhakars/gate-fa4-sm8x-test-utils
Open

Gate FA4 and stabilize attention test imports#3341
sudhakarsingh27 wants to merge 1 commit into
NVIDIA:mainfrom
sudhakarsingh27:sudhakars/gate-fa4-sm8x-test-utils

Conversation

@sudhakarsingh27

Copy link
Copy Markdown
Member

FA4 can be installed on SM8x even though its current implementation rejects those GPUs. Disable selection and skip dedicated FA4 tests there so A100 and L40 use supported attention backends.

FA4 and CUTLASS can also expose a generic utils package on sys.path. Prepend the Transformer Engine test helper directory in the context-parallel test so collection resolves the intended utilities.

Description

Please include a brief summary of the changes, relevant motivation and context.

Fixes # (issue)

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

Please list the changes introduced in this PR:

  • Change A
  • Change B

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

FA4 can be installed on SM8x even though its current implementation rejects those GPUs. Disable selection and skip dedicated FA4 tests there so A100 and L40 use supported attention backends.

FA4 and CUTLASS can also expose a generic utils package on sys.path. Prepend the Transformer Engine test helper directory in the context-parallel test so collection resolves the intended utilities.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
@sudhakarsingh27 sudhakarsingh27 self-assigned this Aug 11, 2026
@greptile-apps

greptile-apps Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR prevents FlashAttention 4 from being selected or explicitly tested on unsupported SM8x GPUs and stabilizes context-parallel test imports against conflicting top-level utils packages.

  • Raises the runtime FA4 architecture gate from SM80 to SM90.
  • Reuses a shared FA4 test marker that skips FA4 suites below SM90.
  • Prepends the Transformer Engine PyTorch test-helper directory before importing utils.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete changed-code defect identified.

The runtime and dedicated-test capability gates consistently exclude SM8x from FA4, while the import-order change makes the intended local test utilities resolve before conflicting installed packages.

Important Files Changed

Filename Overview
transformer_engine/pytorch/attention/dot_product_attention/utils.py Aligns backend selection with FA4's current lack of SM8x support while preserving existing Hopper and newer selection logic.
tests/pytorch/attention/test_attention.py Centralizes the FA4 prerequisite marker and consistently skips dedicated FA4 tests on unsupported SM8x devices.
tests/pytorch/attention/test_attention_with_cp.py Gives the repository's test helper precedence over conflicting installed top-level utils packages.

Reviews (1): Last reviewed commit: "Gate FA4 and stabilize attention test im..." | Re-trigger Greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant