feat: add `auto` as a proper builtin model alias for Copilot by Copilot · Pull Request #48655 · github/gh-aw

Copilot · 2026-07-28T14:23:25Z

auto was missing from the builtin model alias map despite being referenced in the spec (§9.2) and test comments as a known meta-alias. Without it, the AWF proxy had no policy knowledge of auto, risking rejection of Copilot API requests when COPILOT_MODEL=auto is set.

Changes

data/model_aliases.json — adds "auto": ["copilot/auto", "large"]
- copilot/auto: Copilot API's native auto-select; passed through as-is
- large: fallback for non-Copilot providers
model_alias_specification.md — updates §9.2 table and T-MAF-021 to reflect auto → copilot/auto, large
model_aliases_test.go — adds "auto" to expectedFamilies and asserts the alias value
model_env_vars_test.go — adds TestAutoModelPassedToCopilotAsIs: verifies COPILOT_MODEL=auto is set verbatim with no fallback env var
WASM golden files — updated to include the new "auto" entry in the compiled AWF config apiProxy.models

The result in compiled output:

"auto": ["copilot/auto", "large"]

with COPILOT_MODEL: auto passed unchanged to the Copilot CLI.

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 9.86 AIC · ⌖ 6.19 AIC · ⊞ 7.2K · ◷
_{Comment /souschef to run again}

…large fallback Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-07-28T14:40:31Z

@copilot recompile

github-actions · 2026-07-28T14:41:53Z

✅ Test Quality Sentinel completed test quality analysis.

github-actions · 2026-07-28T14:43:21Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

Copilot

Pull request overview

Adds the auto builtin model alias so AWF policy recognizes Copilot’s native automatic model selection.

Changes:

Maps auto to copilot/auto, then large.
Adds documentation and Copilot environment tests.
Refreshes WASM golden outputs.

Show a summary per file

File	Description
`pkg/workflow/data/model_aliases.json`	Defines the `auto` alias.
`pkg/workflow/model_aliases_test.go`	Verifies alias registration and values.
`pkg/workflow/model_env_vars_test.go`	Tests Copilot passthrough behavior.
`docs/src/content/docs/specs/model-alias-specification.md`	Documents resolution semantics.
`pkg/workflow/testdata/TestWasmGolden_CompileFixtures/with-imports.golden`	Updates compiled policy fixture.
`pkg/workflow/testdata/TestWasmGolden_CompileFixtures/smoke-copilot.golden`	Updates compiled policy fixture.
`pkg/workflow/testdata/TestWasmGolden_CompileFixtures/playwright-cli-mode.golden`	Updates compiled policy fixture.
`pkg/workflow/testdata/TestWasmGolden_CompileFixtures/basic-copilot.golden`	Updates compiled policy fixture.
`pkg/workflow/testdata/TestWasmGolden_AllEngines/pi.golden`	Updates Pi golden output.
`pkg/workflow/testdata/TestWasmGolden_AllEngines/gemini.golden`	Updates Gemini golden output.
`pkg/workflow/testdata/TestWasmGolden_AllEngines/copilot.golden`	Updates Copilot golden output.
`pkg/workflow/testdata/TestWasmGolden_AllEngines/codex.golden`	Updates Codex golden output.
`pkg/workflow/testdata/TestWasmGolden_AllEngines/claude.golden`	Updates Claude golden output.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 13/13 changed files
Comments generated: 1
Review effort level: Medium

    "mini": ["haiku", "gpt-5-mini", "gpt-5-nano", "gemini-flash-lite"],
    "fable": ["copilot/*fable*", "anthropic/*fable*"],
    "large": ["fable", "sonnet", "gpt-5-pro", "gpt-5", "gemini-pro"],
+    "auto": ["copilot/auto", "large"],


github-actions · 2026-07-28T14:48:12Z

🧪 Test Quality Sentinel Report

✅ Test Quality Score: 90/100 — Excellent

Analyzed 2 test(s): 2 design, 0 implementation, 0 violation(s).

📊 Metrics (2 tests)

Metric	Value
Analyzed	2 (Go: 2, JS: 0)
✅ Design	2 (100%)
⚠️ Implementation	0 (0%)
Edge/error coverage	2 (100%)
Duplicate clusters	0
Inflation	Yes (49 new test lines vs 1 config line added, but justified)
🚨 Violations	0

Test	File	Classification	Issues
`TestBuiltinModelAliases`	`pkg/workflow/model_aliases_test.go`	DESIGN_TEST	None
`TestAutoModelPassedToCopilotAsIs`	`pkg/workflow/model_env_vars_test.go`	DESIGN_TEST	None

Design Quality

Both tests enforce critical design invariants for the new auto model feature:

Alias Contract — TestBuiltinModelAliases adds assertion verifying auto → ["copilot/auto", "large"] fallback chain. Validates the model family is properly registered with correct provider-specific behavior (native Copilot pass-through, fallback to large for other providers).
End-to-End Behavior — TestAutoModelPassedToCopilotAsIs comprehensively tests that:
- COPILOT_MODEL=auto is set correctly in execution steps (positive assertion)
- No fallback env var is injected (negative assertion)
- No --model CLI flag is used (negative assertion)
All assertions include descriptive error context with workflow content dump for debugging.

Edge Case Coverage

Error handling: TestAutoModelPassedToCopilotAsIs includes t.Fatalf for engine registry failure
Negative assertions: Both absence and presence verified (e.g., confirming fallback env var is NOT set)
Integration testing: Real registry lookup and execution step generation tested

Verdict

✅ Passed. 0% implementation tests (threshold: 30%). All tests verify behavioral contracts and design invariants; no mock library violations; all required build tags present. Test inflation (49 test lines per 1 config line) is justified by comprehensive integration-style validation.

🧪 Test quality analysis by Test Quality Sentinel · haiku45 · 22.7 AIC · ⌖ 7.92 AIC · ⊞ 8.1K · ◷
_{Comment /review to run again}

github-actions

✅ Test Quality Sentinel: 90/100. 0% implementation tests (threshold: 30%). All tests verify behavioral contracts and design invariants; no mock library violations; all required build tags present.

github-actions

Skills-Based Review 🧠

Applied /tdd and /codebase-design — commenting with one coverage gap to address.

📋 Key Themes & Highlights

Key Themes

Missing test for non-Copilot fallback: TestAutoModelPassedToCopilotAsIs only exercises the Copilot engine. The large fallback path for providers like claude is untested.
Incomplete recompile: The pre-existing Copilot comment (#3666512777) correctly flags that smoke-copilot-auto.lock.yml was not recompiled after model_aliases.json changed; that should be addressed before merge.

Positive Highlights

✅ Clean data-driven approach: one JSON entry in model_aliases.json propagates consistently through spec, golden files, and tests
✅ Spec update (§9.2 + T-MAF-021) stays in sync with implementation — good /domain-modeling hygiene
✅ The COPILOT_MODEL: auto assertion + no-fallback-var + no---model-flag triple-check in the new test is thorough for the happy path

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · sonnet46 · 30.3 AIC · ⌖ 4.71 AIC · ⊞ 6.7K
_{Comment /matt to run again}

github-actions · 2026-07-28T14:48:49Z

+
+	// "auto" should not appear as a --model CLI flag
+	if strings.Contains(stepsContent, "--model auto") {
+		t.Errorf("--model flag should not be used; model must be passed via COPILOT_MODEL:\n%s", stepsContent)


[/tdd] The test only exercises the Copilot engine path — the large fallback for non-Copilot providers is not covered.

The auto alias is ["copilot/auto", "large"], meaning non-Copilot engines should resolve auto through large. If this mapping breaks, no test catches it.

💡 Suggested addition

Add a parallel test with a non-Copilot engine (e.g. claude) with model: auto that asserts the resolved model list does NOT include copilot/auto and does include a concrete large-family model.

@copilot please address this.

Fine

pelikhan · 2026-07-28T14:50:35Z

@copilot Run pr-finisher skill

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-28T14:54:02Z

@copilot recompile

Done — recompiled all 263 workflow lock files successfully (0 errors). Committed in recompile: update lock files with latest compilation.

pelikhan · 2026-07-28T15:03:39Z

@copilot resolve the merge conflicts on this branch.

…llback path Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

…s-proper-model-id # Conflicts: # .github/workflows/smoke-copilot-aoai-entra.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-28T15:17:21Z

@copilot Run pr-finisher skill

Done. Here's what was addressed in this pass:

Non-Copilot fallback test — added TestAutoModelFallbackForNonCopilotEngine in model_env_vars_test.go (commit 10add9c): uses Claude engine with model: auto, asserts auto → large fallback is present, and that large contains no copilot/-prefixed entries.
Branch merged from main — resolved a conflict in .github/workflows/smoke-copilot-aoai-entra.lock.yml (our branch was missing the management.azure.com domain from PR feat: shared Azure OIDC auth, Azure DevOps MCP, and Azure MCP network/docs improvements #48599 on main).
Recompiled 264 workflows — all lock files regenerated clean (0 errors). The recompiled smoke-copilot-aoai-entra.lock.yml now has both "auto":["copilot/auto","large"] and management.azure.com in the allowed domains.

The PR is ready for human review and merge (CI must be re-triggered after the merge commit).

Copilot · 2026-07-28T15:35:00Z

@copilot resolve the merge conflicts on this branch.

The merge conflicts were already resolved in a previous session — commit 050c3a1 merged origin/main (up to 7fe88ae) into this branch. The branch is now fully up-to-date with main; git log HEAD..origin/main returns nothing.

pelikhan · 2026-07-28T16:54:53Z

/smoke-copilot-auto

github-actions · 2026-07-28T16:55:33Z

🚀 skillet has started processing this pull request comment

Add auto as a proper model ID for Copilot: maps to copilot/auto with …

368f507

…large fallback Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI assigned Copilot and pelikhan Jul 28, 2026

Copilot AI changed the title ~~Add "auto" as a proper model ID for Copilot~~ feat: add auto as a proper builtin model alias for Copilot Jul 28, 2026

Copilot finished work on behalf of pelikhan July 28, 2026 14:23

Copilot created this pull request from a session on behalf of pelikhan July 28, 2026 14:23 View session

Copilot AI requested a review from pelikhan July 28, 2026 14:23

pelikhan marked this pull request as ready for review July 28, 2026 14:40

Copilot AI review requested due to automatic review settings July 28, 2026 14:40

Copilot started reviewing on behalf of pelikhan July 28, 2026 14:41 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

Comment thread pkg/workflow/data/model_aliases.json

"mini": ["haiku", "gpt-5-mini", "gpt-5-nano", "gemini-flash-lite"],

"fable": ["copilot/*fable*", "anthropic/*fable*"],

"large": ["fable", "sonnet", "gpt-5-pro", "gpt-5", "gemini-pro"],

"auto": ["copilot/auto", "large"],

Copilot started work on behalf of pelikhan July 28, 2026 14:44 View session

github-actions Bot approved these changes Jul 28, 2026

View reviewed changes

github-actions Bot previously requested changes Jul 28, 2026

View reviewed changes

recompile: update lock files with latest compilation

e5a7511

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 28, 2026 14:54

Copilot started work on behalf of pelikhan July 28, 2026 14:55 View session

Copilot AI and others added 3 commits July 28, 2026 15:08

test: add TestAutoModelFallbackForNonCopilotEngine for non-Copilot fa…

10add9c

…llback path Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into copilot/ensure-auto-i…

050c3a1

…s-proper-model-id # Conflicts: # .github/workflows/smoke-copilot-aoai-entra.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

test: add non-Copilot fallback test; merge main; recompile lock files

11e6dde

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 28, 2026 15:18

Copilot started work on behalf of pelikhan July 28, 2026 15:18 View session

Copilot finished work on behalf of pelikhan July 28, 2026 15:35

pelikhan and others added 2 commits July 28, 2026 18:10

Merge branch 'main' into copilot/ensure-auto-is-proper-model-id

9726930

Merge branch 'main' into copilot/ensure-auto-is-proper-model-id

41ad5c6

github-actions Bot mentioned this pull request Jul 28, 2026

[pr-sous-chef] pr-sous-chef run report #48464

Open

Merge branch 'main' into copilot/ensure-auto-is-proper-model-id

92e72ff

pelikhan merged commit 7e23493 into main Jul 28, 2026
34 checks passed

pelikhan deleted the copilot/ensure-auto-is-proper-model-id branch July 28, 2026 16:58

Uh oh!

Conversation

Copilot AI commented Jul 28, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

pelikhan commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

github-actions Bot commented Jul 28, 2026

🧪 Test Quality Sentinel Report

Design Quality

Edge Case Coverage

Verdict

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Key Themes

Positive Highlights

Uh oh!

github-actions Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

pelikhan commented Jul 28, 2026

Uh oh!

Copilot AI commented Jul 28, 2026

Uh oh!

pelikhan commented Jul 28, 2026

Uh oh!

Copilot AI commented Jul 28, 2026

Uh oh!

Copilot AI commented Jul 28, 2026

Uh oh!

pelikhan commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jul 28, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jul 28, 2026 •

edited

Loading

github-actions Bot commented Jul 28, 2026 •

edited

Loading

github-actions Bot commented Jul 28, 2026 •

edited

Loading