evals: promote agent-mount contract to v4-spike

[miguelg719]

Summary

Corrects the base-target mistake from #2590 by landing its merged agent-mount contract on v4-spike.

This branch is exactly current v4-spike plus the #2590 squash commit:

• AgentMount is the lifecycle-free agent delivery binding
• ToolStartResult owns the single runtime lifecycle and captureEvidence
• contract tests cover independent surface/mount modalities

After merge, #2591 and the remaining stack will be rebased directly onto v4-spike.

Verification

• pnpm --filter @browserbasehq/stagehand-evals typecheck
• contract and downstream stack suites validated during restack

---

Summary by cubic

Promotes the agent-mount tool-surface contract to v4-spike, adding type-safe agent delivery bindings and best-effort evidence capture, independent of CoreTool.surface. Types and tests only; no runtime changes.

• New Features
  • ToolStartResult adds optional agentMount and captureEvidence(): Promise<ProbeEvidence>.
  • New AgentMount union (via: handles | mcp | cli) with promptInstructions; plus AgentRunToolSpec.
  • Constants: AGENT_RUN_TOOL_SERVER, AGENT_RUN_TOOL_NAME, AGENT_RUN_TOOL_RESERVED_HANDLES. Enums: ToolSurface adds stagehand_code; CoreTool.family adds stagehand.
  • Tests (packages/evals/tests/core/tool-contract.test.ts) validate surface/delivery independence, handle mounts, and evidence capture.

^{Written for commit fd74210. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: fd74210

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

1 issue found across 2 files

Confidence score: 3/5

• In packages/evals/core/contracts/tool.ts, stagehand_code was added to ToolSurface but runtime resolvers still fall through to "not implemented"/"No default startup profile" paths, so selecting that surface can fail at runtime instead of executing a tool flow — align resolver/default-profile handling with the updated union and add a guard test for stagehand_code.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/core/contracts/tool.ts">

<violation number="1" location="packages/evals/core/contracts/tool.ts:9">
P2: `stagehand_code` is now part of `ToolSurface`, but the runtime resolvers still don't handle it, so selecting that surface reaches thrown "not implemented"/"No default startup profile" paths. Keeping the union aligned with implemented registry/startup branches (or deferring this union member) would avoid a misleading supported-surface contract.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant Harn as Agent Harness
    participant ToolCtrl as CoreTool
    participant StartRes as ToolStartResult
    participant Mount as AgentMount
    participant Evid as captureEvidence

    Note over Harn,Evid: NEW: Agent-Mount Contract Flow (v4-spike)

    Harn->>ToolCtrl: start(input) via CoreTool interface
    activate ToolCtrl
    ToolCtrl-->>StartRes: Return ToolStartResult
    deactivate ToolCtrl

    Note over StartRes: Contains optional agentMount + captureEvidence

    alt Native surface (no mount)
        Harn->>StartRes: Use session directly
        StartRes-->>Harn: Cleanup lifecycle
    else Agent-mounted delivery
        Harn->>Mount: Access agentMount via
        alt via: "handles"
            Mount->>Mount: Resolve handles (Record<string, unknown>)
            Mount->>Mount: Validate reserved handles (startUrl, task, console)
        else via: "mcp"
            Mount->>Mount: Load mcpServers config
        else via: "cli"
            Mount->>Mount: Build command (bin, args, cwd, env)
        end
        Harn->>StartRes: Use promptInstructions for agent context
    end

    opt Evidence capture requested
        Harn->>Evid: Call captureEvidence()
        activate Evid
        Evid->>Evid: Swallow per-field failures, never throw
        Evid-->>Harn: Return ProbeEvidence (screenshot, url, ariaTree)
        deactivate Evid
    end

    Harn->>StartRes: Call cleanup()
    activate StartRes
    Note over StartRes: captureEvidence invalid after this
    StartRes-->>Harn: Runtime released
    deactivate StartRes

    alt Compile-time validation
        Harn->>Mount: Verify discriminated union fields
        Mount-->>Harn: TypeScript ensures via-specific properties
        Note over Harn: e.g., CLI mount requires command field
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>