feat: error when agent-job step outputs are referenced in the prompt body

docs/adr/48610-validate-agent-step-output-in-prompt.md

@@ -0,0 +1,48 @@
+# ADR-48610: Compile-Time Validation of Agent-Job Step Outputs in Prompt Body
+
+**Date**: 2026-07-28
+**Status**: Draft
+**Deciders**: pelikhan, copilot-swe-agent
+
+---
+
+### Context
+
+gh-aw workflows have a two-job architecture: the **activation job** renders the prompt (markdown body) and starts the agent; the **agent job** runs the steps defined in `steps:`, `pre-steps:`, `pre-agent-steps:`, and `post-steps:` sections. Because the prompt is rendered before the agent job executes, any `${{ steps.STEP_ID.outputs.* }}` expression referencing an agent-job step silently resolves to an empty string at prompt-creation time. This is a confusing silent failure: the workflow compiles successfully, the agent receives an incomplete prompt, and the root cause is non-obvious. Authors familiar with standard GitHub Actions expect step outputs to be available in surrounding content, and this architectural constraint is not surfaced until runtime (if at all).
+
+### Decision
+
+We will add a compile-time validator (`validateStepsOutputsNotInPrompt`) that scans the prompt body for `${{ steps.STEP_ID.* }}` expressions and cross-references the step ID against the set of IDs declared in agent-job step sections. If a match is found, compilation is rejected with an actionable error that names the offending step and directs authors to the file-based alternative (`/tmp/gh-aw/agent/result.txt`). Built-in activation-job steps (e.g., `steps.sanitized`) are not flagged because they do not appear in user-visible step lists. The validator is wired into the existing `validateExpressions` gate in `compiler_validators.go`.
+
+### Alternatives Considered
+
+#### Alternative 1: Runtime Warning Without Compilation Failure
+
+Detect the pattern at compile time but emit only a log warning rather than a hard error, allowing the workflow to proceed. Authors would see a warning in CI logs but the workflow would still run with an empty prompt value.
+
+This was rejected because a warning is too easy to overlook, and the resulting runtime behavior (silently empty prompt variable) is indistinguishable from an intentional empty value. The goal is to prevent a class of hard-to-debug bugs, which requires failing loudly at the earliest opportunity.
+
+#### Alternative 2: Documentation Only, No Enforcement
+
+Document the two-job execution order in the workflow authoring guide and rely on authors to internalize the constraint without automated enforcement.
+
+This was rejected because it scales poorly: each new author must discover the limitation individually (typically after a confusing empty-prompt incident), and there is no mechanism to prevent the pattern from being introduced by automated tooling or copy-paste from standard GitHub Actions workflows.
+
+### Consequences
+
+#### Positive
+- Prevents the silent empty-prompt class of bugs at the earliest point in the authoring cycle (compile time), before the workflow is ever run.
+- Provides an actionable error message that names the offending step ID and links directly to the file-based alternative, reducing debugging time.
+- Makes the two-job execution boundary explicit in the toolchain, reinforcing the architectural constraint for all authors.
+
+#### Negative
+- Any workflow that currently relies on the silent-empty behavior (however unintentionally) will fail compilation after this change is deployed, requiring a migration to file-based data passing.
+- The YAML parsing step in `addStepIDsFromYAML` adds a small compile-time cost for every workflow that contains step sections, even when the prompt contains no `${{ steps.* }}` expressions (mitigated by an early-exit fast path).
+
+#### Neutral
+- The validator only flags agent-job step IDs that appear in the user-visible step YAML sections; built-in activation-job step references (e.g., `${{ steps.sanitized.outputs.text }}`) pass through unchanged, so existing valid workflows are unaffected.
+- The file-based pattern (`/tmp/gh-aw/agent/result.txt`) that replaces the forbidden pattern was already the recommended approach; this change codifies that recommendation as an enforcement rule.
+
+---
+
+*ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.*

pkg/workflow/compiler_validators.go

@@ -62,6 +62,13 @@ func (c *Compiler) validateExpressions(workflowData *WorkflowData, markdownPath
 	// of the recommended /tmp/gh-aw/agent/ subtree.
 	c.validatePromptTmpPaths(workflowData, markdownPath)
 
+	// Detect agent-job step outputs referenced in the prompt. The prompt is rendered
+	// in the activation job; agent-job steps run later and their outputs are not
+	// available at prompt-creation time.
+	if err := validateStepsOutputsNotInPrompt(workflowData); err != nil {
+		return formatCompilerError(markdownPath, "error", err.Error(), err)
+	}
+
 	return nil
 }
 

pkg/workflow/steps_output_in_prompt_validation.go

@@ -0,0 +1,133 @@
+// This file validates that agent-job step outputs are not referenced in the
+// workflow prompt (markdown body). The prompt is built in the activation job;
+// agent-job steps run after the activation job and their outputs are therefore
+// unavailable when the prompt is assembled. Steps that need to pass data into
+// the prompt should write results to files instead.
+
+package workflow
+
+import (
+	"fmt"
+	"regexp"
+	"strings"
+
+	"github.com/github/gh-aw/pkg/logger"
+	"github.com/goccy/go-yaml"
+)
+
+var stepsOutputInPromptLog = logger.New("workflow:steps_output_in_prompt_validation")
+
+// stepsExprBodyPattern extracts the body of each ${{ ... }} expression so that
+// all operands can be inspected for steps references.
+var stepsExprBodyPattern = regexp.MustCompile(`\$\{\{([\s\S]*?)\}\}`)
+
+// stepsRefInExprPattern matches every steps.STEP_ID occurrence inside an
+// expression body (after stripping the surrounding ${{ }}).
+var stepsRefInExprPattern = regexp.MustCompile(`\bsteps\.([a-zA-Z0-9_-]+)`)
+
+// validateStepsOutputsNotInPrompt checks that no agent-job step outputs are
+// referenced in the prompt body (MarkdownContent). The prompt is rendered in
+// the activation job; steps defined in the frontmatter steps: / pre-steps: /
+// pre-agent-steps: / post-steps: sections run inside the agent job (after
+// activation) and their outputs are not available at prompt-creation time.
+//
+// Use files (e.g. /tmp/gh-aw/agent/result.txt) to pass data from agent-job
+// steps into the prompt instead of step outputs.
+func validateStepsOutputsNotInPrompt(workflowData *WorkflowData) error {
+	if workflowData == nil || workflowData.MarkdownContent == "" {
+		return nil
+	}
+	if !strings.Contains(workflowData.MarkdownContent, "${{ steps.") {
+		return nil
+	}
+
+	agentStepIDs := extractAgentJobStepIDs(workflowData)
+	if len(agentStepIDs) == 0 {
+		stepsOutputInPromptLog.Print("No agent-job step IDs found; skipping prompt step-output check")
+		return nil
+	}
+
+	seen := make(map[string]struct{})
+	var offending []string
+	for _, exprMatch := range stepsExprBodyPattern.FindAllStringSubmatch(workflowData.MarkdownContent, -1) {
+		if len(exprMatch) < 2 {
+			continue
+		}
+		body := exprMatch[1]
+		for _, refMatch := range stepsRefInExprPattern.FindAllStringSubmatch(body, -1) {
+			if len(refMatch) < 2 {
+				continue
+			}
+			stepID := refMatch[1]
+			if _, isAgentStep := agentStepIDs[stepID]; isAgentStep {
+				if _, already := seen[stepID]; !already {
+					offending = append(offending, stepID)
+					seen[stepID] = struct{}{}
+				}
+			}
+		}
+	}
+
+	if len(offending) == 0 {
+		stepsOutputInPromptLog.Print("No agent-job step outputs found in prompt")
+		return nil
+	}
+
+	stepsOutputInPromptLog.Printf("Found %d agent-job step output(s) in prompt: %v", len(offending), offending)
+
+	return NewValidationError(
+		"steps-output-in-prompt",
+		"steps referenced in prompt: "+strings.Join(offending, ", "),
+		fmt.Sprintf(
+			"step(s) %s are defined in the agent job and run after the prompt is rendered in the activation job; "+
+				"their outputs are not available at prompt-creation time",
+			strings.Join(offending, ", "),
+		),
+		"Write step results to a file (e.g. /tmp/gh-aw/agent/result.txt) and reference that file path in the prompt instead of using ${{ steps.STEP_ID.outputs.* }}.",
+	)
+}
+
+// extractAgentJobStepIDs returns the set of step IDs defined across all
+// agent-job step sections: steps:, pre-steps:, pre-agent-steps:, post-steps:.
+func extractAgentJobStepIDs(workflowData *WorkflowData) map[string]struct{} {
+	ids := make(map[string]struct{})
+	for _, stepsYAML := range []string{
+		workflowData.CustomSteps,
+		workflowData.PreSteps,
+		workflowData.PreAgentSteps,
+		workflowData.PostSteps,
+	} {
+		if stepsYAML == "" {
+			continue
+		}
+		addStepIDsFromYAML(stepsYAML, ids)
+	}
+	return ids
+}
+
+// addStepIDsFromYAML parses a YAML string that wraps a steps list under a
+// top-level key (e.g. "steps:", "pre-steps:") and collects each step's id.
+func addStepIDsFromYAML(stepsYAML string, ids map[string]struct{}) {
+	var wrapper map[string]any
+	if err := yaml.Unmarshal([]byte(stepsYAML), &wrapper); err != nil {
+		stepsOutputInPromptLog.Printf("Failed to parse steps YAML: %v", err)
+		return
+	}
+	// The root key varies ("steps", "pre-steps", "pre-agent-steps", "post-steps"),
+	// so iterate over all top-level values and look for step lists.
+	for _, value := range wrapper {
+		steps, ok := value.([]any)
+		if !ok {
+			continue
+		}
+		for _, step := range steps {
+			stepMap, ok := step.(map[string]any)
+			if !ok {
+				continue
+			}
+			if id, ok := stepMap["id"].(string); ok && id != "" {
+				ids[id] = struct{}{}
+			}
+		}
+	}
+}