Skip to content

emrg: the POSIX-arm guard treats any pytest.skip as a gate, so a which-only shell gate passed and test-windows went red #2348

Closed

Description

@how2how2how2-arch

What happened

tests/test_posix_arms_are_gated.py exists to catch an arm that reaches a POSIX-only primitive without a gate. On 2026-10-17 two new arms in tests/test_evolution_prompt_prompt_drift.py started a POSIX shell with a gate of

if shutil.which("bash") is None:
    pytest.skip("no bash on this host")

— the exact spelling the guard's own module docstring says is not the guard, because on windows-2025 a bare bash resolves to the WSL launcher C:\Windows\System32\bash.exe, which exists, cannot run a script, and answers "Windows Subsystem for Linux has no installed distributions."

The sweep called both arms GATED, CI agreed with the arms instead: test-windows failed on run 39027157421 (test_every_shell_block_parses, test_the_step_zero_line_finds_a_file_that_changed) while test passed — the one-leg red the file exists to prevent.

Why the guard missed it

_is_gated ends with

return "pytest.skip(" in _body_source(node)

so any pytest.skip(...) anywhere in a test body counts as a gate, whatever condition it sits under. The comment above it says a skip under no platform condition "is a red on both legs rather than the one-leg red this file exists for" — and that premise is false for exactly this shape, which is green on every POSIX host and red on the Windows leg.

Measurement, both directions

The guard's own _index over the head that went red (76cf63ad):

guard as shipped fallback removed
test_every_shell_block_parses GATED UNGATED
test_the_step_zero_line_finds_a_file_that_changed GATED UNGATED

and with the fallback removed over the whole suite as it stands (after the arms were given a real gate): 0 arms lose their gate — nothing in this tree relies on that fallback, so removing it is not a rewrite of a hundred call sites.

Fix

Remove the fallback, so a pytest.skip counts only when its condition names the platform (a skipif condition, an if test, a module or class marker, or a module-level pytestmark), and add a control whose synthetic arm carries exactly the which("bash") is None skip and must be reported ungated. The arms themselves are fixed in #2343 — they now carry a marker that asks the shell to run one command (bash -c 'echo ok') before trusting the name.

Activity

  1. how2how2how2-arch commented on Oct 17, 2026

    @how2how2how2-arch
    CollaboratorAuthor

    Handled by #2350

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions