[Python][tests] E2E harness picks the first @github/copilot-* package instead of the current platform package

[syedkazmi14]

Summary

The Python test harness resolves the Copilot CLI by scanning nodejs/node_modules/@github for copilot-* directories and returning the first index.js it finds:

for platform_pkg in sorted(github_modules.glob("copilot-*")):
    candidate = platform_pkg / "index.js"
    if candidate.exists():
        return str(candidate.resolve())

That selection is based on directory order, not the current OS/arch. As a result, Python tests can choose the wrong platform package when multiple @github/copilot-* packages are present, or whenever the checked-out node_modules tree contains only a mismatched platform package.

This logic lives in python/e2e/testharness/context.py and also affects tests that import CLI_PATH from that harness, such as python/test_client.py.

Why this is a bug

The Node SDK already resolves platform packages explicitly in nodejs/src/client.ts via getCliPlatformPackageNames(), using the current platform/arch and handling Linux linux vs linuxmusl variants.

The repo also already fixed the same class of problem for the .NET E2E harness in #2093, where directory enumeration order was selecting the wrong platform-specific package.

Python still has the older directory-order-based logic.

Repro

1. Leave COPILOT_CLI_PATH unset.
2. Have more than one @github/copilot-* package under nodejs/node_modules/@github, or otherwise have a checked-out node_modules tree whose available copilot-* package(s) do not match the current host platform.
3. Run any Python test that imports e2e.testharness.context or CLI_PATH.
4. get_cli_path_for_tests() returns the first matching package instead of the package for the current platform.
5. The Python test process then tries to launch the wrong CLI entrypoint.

Expected behavior

The Python harness should resolve the platform package the same way the runtime code already does elsewhere in the repo:

• choose win32 / darwin / linux based on the current platform
• choose the correct architecture
• on Linux, distinguish linux vs linuxmusl

Suggested fix

Mirror the platform-aware package selection used in nodejs/src/client.ts instead of scanning copilot-* in sorted order.

Related

• Fix active .NET and Python CI failures #2093 fixes the analogous .NET test harness issue
• Update @github/copilot to 1.0.64-1 #1739 introduced the newer @github/copilot package layout where the runnable index.js lives in platform-specific packages

[github-actions]

Investigation Findings

I investigated this issue by examining python/e2e/testharness/context.py and nodejs/src/client.ts.

What I found

The bug is confirmed. In python/e2e/testharness/context.py (lines 36–39), the get_cli_path_for_tests() function resolves the CLI path by iterating over @github/copilot-* directories in sorted (alphabetical) order and returning the first one that contains an index.js:

for platform_pkg in sorted(github_modules.glob("copilot-*")):
    candidate = platform_pkg / "index.js"
    if candidate.exists():
        return str(candidate.resolve())

This means that when multiple platform packages are present (e.g., both copilot-darwin-arm64 and copilot-linux-x64), or when the available package does not match the current host OS/arch, the harness silently picks the wrong binary, causing tests to fail with a mismatched CLI entrypoint.

How it should work

The Node SDK already has the correct platform-aware resolution in nodejs/src/client.ts via getCliPlatformPackageNames(), which:

• Maps process.platform to win32 / darwin / linux (or linuxmusl for musl libc)
• Appends the current architecture (process.arch)
• Returns ordered candidates like [@github/copilot-linux-x64, @github/copilot-linuxmusl-x64]

The Python harness should mirror this logic using platform.system() and platform.machine() (plus a musl-vs-glibc check on Linux), matching the fix already applied to the .NET harness in #2093.

Conclusion

Labeling as bug — the current behavior is unintentional and can silently run the wrong CLI binary during E2E tests.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Bug Handler for #2103 · 21.8 AIC · ⌖ 4.97 AIC · ⊞ 4.5K · ◷