Report eval results in OpenTelemetry spans

[Copilot]

BinEval results were persisted as artifacts but absent from OTLP telemetry, preventing direct evaluation analysis in observability backends.

• Summary metrics
  • Add gh-aw.evals.count, yes, no, and unknown attributes to eval-job conclusion spans.
• Per-result events
  • Emit an eval.result event with evaluation ID, normalized answer, and model.
• Data safety
  • Exclude free-form question text from telemetry.
  • Ignore malformed JSONL records and normalize unsupported answers to UNKNOWN.
• Release
  • Add a patch changeset.

[Copilot]

🟡 Changes recommended

Answer normalization can produce telemetry inconsistent with persisted-result analysis.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds BinEval summaries and result events to eval-job OpenTelemetry conclusion spans while excluding question text.

Changes:

• Emits normalized eval counts and per-result events.
• Adds telemetry tests and a patch changeset.
• Reorders a workflow permission schema entry.

File summaries

File	Description
actions/setup/js/send_otlp_span.cjs	Builds and attaches eval telemetry.
actions/setup/js/send_otlp_span.test.cjs	Tests telemetry content and job scoping.
pkg/workflow/schemas/github-workflow.json	Reorders a permission definition.
.changeset/report-evals-otel.md	Records the patch release note.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[gh-aw-bot]

@copilot this PR has open review feedback and the branch could use a refresh. Please address the latest review comments, update the branch with the latest base changes if needed, and run the pr-finisher skill before handing back for maintainer review.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 13.2 AIC · ⌖ 8.63 AIC · ⊞ 9.2K · ◷
_{Comment /souschef to run again}

[Copilot]

@copilot this PR has open review feedback and the branch could use a refresh. Please address the latest review comments, update the branch with the latest base changes if needed, and run the `pr-finis...

Addressed the review feedback in 1f89c7e: eval answers now trim surrounding whitespace before normalization, with a padded-answer regression test. The branch is already current with main; completing the requested PR-finisher validation now.

[gh-aw-bot]

@copilot this PR still has open review feedback and the branch could use a refresh. Please address the latest review comments, update the branch with the latest base changes if needed, and run the pr-finisher skill before handing back for maintainer review.

• Open review: Report eval results in OpenTelemetry spans #57166 (review)
• Run for context: https://github.com/github/gh-aw/actions/runs/33314176961

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 11 AIC · ⌖ 8.63 AIC · ⊞ 9.2K · ◷
_{Comment /souschef to run again}

[Copilot]

@copilot this PR still has open review feedback and the branch could use a refresh. Please address the latest review comments, update the branch with the latest base changes if needed, and run the `pr...

Confirmed the linked review feedback is resolved, and the branch is current with main (4 ahead, 0 behind). The referenced PR Sous Chef run completed successfully; its firewall notice is workflow-run telemetry, not an actionable change in this PR. Local format, lint, and unit validation pass.