diff --git a/CHANGELOG.md b/CHANGELOG.md index f7b2570..4076249 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,87 @@ # Changelog +## Unreleased + +- AX reports when a window snapshot returns only its top-level root. + The diagnostic asks for a screenshot and verification of prior action outcomes + before retrying. It does not infer a renderer crash, reload the application, + or replay an action. Reads with coverage gaps say that omitted content may + still exist; scoped reads do not trigger the window-level diagnostic. + +- AX `activate` with a window ref or title can switch the target within an + action batch. Later refs and ref-less steps use the selected window; + refs from other windows require another explicit activation. Each step + reports its window, and the final observation uses the last executed window. +- Linux AX `click` and `scroll` accept `allow_global_input`, defaulting to + `true`. Setting it to `false` forbids driver focus/raise and synthesized + pointer/keyboard input, including editor focus preparation and Qt state + synchronization. Semantic actions can still change application focus. + Unsupported uses reject the whole batch with `invalid_argument` before + input. A required global route returns `global_input_forbidden` without + input from that action. Under this policy, an AT-SPI action returning + false after invocation reports `effect_unknown` without a global retry. +- Linux AX tab clicks prefer the parent container's accessibility selection + operation. Supported list and table items without their own action also + use parent selection. Failed selection verification returns + `effect_unknown` without global input fallback. +- Linux AX scroll can use accessible scrollbar ranges with readback. + A page means five minimum range increments, not pixels; pages remain + approximate, and a clamped boundary no-op is verified. AX results retain + synthesis route notes and warnings without claiming successful effects. +- Linux AX `set_value` reads back the requested control after keyboard fallback. + Unconfirmed values return `effect_unknown` and stop the batch, preserving input + evidence and warning against retyping text that may remain in an editor. + +- Linux AX `observe` accepts `read_text` for an existing element `root`. + It pages AT-SPI Text values and complete accessible names or descriptions + in at most 4000 Unicode code points per call. Results include raw text, + offsets, total length, and a next-page call; structured text pages retain + the raw text too. Invalid ranges, owner mismatches, and unavailable Text + fail explicitly. Lower inline budgets reduce page size without raising + tree caps. Each page reads live content; detected length changes fail that + call, but cross-page and same-length edits have no snapshot guarantee. +- Linux AX trees mark clipped fields with their full lengths and read calls. + Non-interactive Text content gets an 80-code-point preview and a bindable + ref. Existing Go cell, find excerpt, name, and description rules remain. +- Text paging and the revised hybrid guide change the model-visible contract + and require a new evaluation attempt. Keep instruction revisions separate; + the interface changes do not establish evaluation performance. +- AX batch errors expose the failed action's reason and a failure-specific + recovery hint while retaining batch progress and input evidence. No-input + argument and unsupported-operation failures use `invalid_argument`; + uncertain input remains `effect_unknown`. +- Invalid AX activation, settle, and double-click calls explain the supported + call shapes without accepting aliases. +- Missing, ambiguous, or moved provider-bound AX observation roots report `stale_ref` and + request a new application observation without the invalid root. + +- Linux AX `set_value` and `select` choose Chromium and Electron HTML select + options through accessibility without opening the popup or activating the + window. Both actions read back the selection and fail explicitly when it + cannot be confirmed. HTML list box mutations are refused because their + accessibility actions do not emit both `input` and `change` events. +- Linux AX and GUI accept case-insensitive keypad and X11 key names. + Named keys use their requested symbol, including keypad digits with NumLock + off. Unknown keys report `invalid_argument` with accepted spelling examples + before the key step takes focus or emits input. +- Linux AX preserves references to controls hidden by an in-window modal dialog. + After the dialog closes, each action still checks the live target; a provider + rebuild requires a unique semantic match in the original runtime path and window. +- AX reports text-value evidence for `type` steps without a ref, including + unchanged or unreadable fields. A focus change alone does not confirm typing. +- Linux AX checks focused-field readback after keyboard typing and reports + unconfirmed text explicitly. If window activation changes the resolved text + field, typing stops before keyboard input. +- Linux lists managed application windows that omit `_NET_WM_PID` when + X-Resource can identify their owner. Windows without an accessibility tree + support AX activation without pointer movement. Unverified owners produce + an explicit inventory issue instead of an actionable reference. +- Linux AX `set_value` and `select` choose native GTK combo box options through + accessibility without opening the popup or taking keyboard focus. Both actions + verify the active item and fail explicitly if selection cannot be confirmed. + Combo boxes with an entry use the same option selection; their entry refs + remain available for free-form text editing. + ## 1.0.0 First open-source release of Hybrid Computer Use. diff --git a/computer-use/internal/contract/contract_describe.go b/computer-use/internal/contract/contract_describe.go index a8f40d4..3eec298 100644 --- a/computer-use/internal/contract/contract_describe.go +++ b/computer-use/internal/contract/contract_describe.go @@ -9,7 +9,7 @@ import ( // tool schemas, the records envelope and the error table below. Adding an // optional field or operation raises the minor number; removing or changing // one raises the major number. -const ContractVersion = "1.0" +const ContractVersion = "1.1" // AXCodeGUIFrameStale is the GUI code for coordinates that refer to a // screenshot that no longer matches the desktop layout. Nothing was sent. @@ -30,18 +30,19 @@ type PublicError struct { } var PublicErrorCodes = map[string]PublicError{ - AXCodeInvalidArgument: {2, "no", "fix the argument named by path; hint is a corrected call"}, - AXCodeWaitTimeout: {3, "maybe", "observe again; completed counts the steps that ran and the message says what the wait saw"}, - AXCodeStaleRef: {4, "no", "observe or find again and use the new ref; refs never rebind"}, - AXCodeGUIFrameStale: {4, "no", "take a new GUI screenshot and plan coordinates from it"}, - AXCodeWindowChanged: {5, "maybe", "observe before retrying; do not repeat the batch blindly"}, - AXCodeEffectUnknown: {5, "maybe", "observe before retrying; input may have taken effect"}, - AXCodeAppUnavailable: {6, "no", "list apps; the application is not running or not accessible"}, - AXCodeAmbiguousWindow: {6, "no", "use a longer title or the wN from apps"}, - AXCodeWindowNotFound: {6, "no", "list apps and use a wN or an exact title"}, - AXCodeNoFocusedWindow: {6, "no", "activate a window first or pass window_ref"}, - AXCodeRuntimeError: {10, "maybe", "retry reads later; observe before retrying input"}, - GUICodePermissionDenied: {6, "no", GUIPermissionHint}, + AXCodeInvalidArgument: {2, "no", "fix the argument named by path; hint is a corrected call"}, + AXCodeWaitTimeout: {3, "maybe", "observe again; completed counts the steps that ran and the message says what the wait saw"}, + AXCodeStaleRef: {4, "no", "observe or find again and use the new ref; refs never rebind"}, + AXCodeGUIFrameStale: {4, "no", "take a new GUI screenshot and plan coordinates from it"}, + AXCodeWindowChanged: {5, "maybe", "observe before retrying; do not repeat the batch blindly"}, + AXCodeEffectUnknown: {5, "maybe", "observe before retrying; input may have taken effect"}, + AXCodeGlobalInputForbidden: {5, "maybe", "the refused action sent no input; check completed and observe before replanning; keep allow_global_input false unless global input is authorized"}, + AXCodeAppUnavailable: {6, "no", "list apps; the application is not running or not accessible"}, + AXCodeAmbiguousWindow: {6, "no", "use a longer title or the wN from apps"}, + AXCodeWindowNotFound: {6, "no", "list apps and use a wN or an exact title"}, + AXCodeNoFocusedWindow: {6, "no", "activate a window first or pass window_ref"}, + AXCodeRuntimeError: {10, "maybe", "retry reads later; observe before retrying input"}, + GUICodePermissionDenied: {6, "no", GUIPermissionHint}, } // DescribeContract is the machine-readable contract: what `schema` prints diff --git a/computer-use/internal/contract/published_values.go b/computer-use/internal/contract/published_values.go index 303603c..e789152 100644 --- a/computer-use/internal/contract/published_values.go +++ b/computer-use/internal/contract/published_values.go @@ -24,23 +24,24 @@ const ( // Public AX error codes. Internal failure codes stay in telemetry; the model // sees only these. const ( - AXCodeInvalidArgument = "invalid_argument" - AXCodeStaleRef = "stale_ref" - AXCodeWindowChanged = "window_changed" - AXCodeWaitTimeout = "wait_timeout" - AXCodeEffectUnknown = "effect_unknown" - AXCodeAppUnavailable = "app_unavailable" - AXCodeRuntimeError = "runtime_error" - AXCodeAmbiguousWindow = "ambiguous_window" - AXCodeWindowNotFound = "window_not_found" - AXCodeNoFocusedWindow = "no_focused_window" - AXModeApps = "apps" - AXDefaultFindPageSize = 20 - AXDefaultWaitTimeout = 5000 - AXPublicErrorCodeKey = "public_error_code" - AXPublicErrorPathKey = "public_error_path" - AXPublicErrorHintKey = "public_error_hint" - AXPublicErrorDetailKey = "public_error_message" + AXCodeInvalidArgument = "invalid_argument" + AXCodeStaleRef = "stale_ref" + AXCodeWindowChanged = "window_changed" + AXCodeWaitTimeout = "wait_timeout" + AXCodeEffectUnknown = "effect_unknown" + AXCodeGlobalInputForbidden = "global_input_forbidden" + AXCodeAppUnavailable = "app_unavailable" + AXCodeRuntimeError = "runtime_error" + AXCodeAmbiguousWindow = "ambiguous_window" + AXCodeWindowNotFound = "window_not_found" + AXCodeNoFocusedWindow = "no_focused_window" + AXModeApps = "apps" + AXDefaultFindPageSize = 20 + AXDefaultWaitTimeout = 5000 + AXPublicErrorCodeKey = "public_error_code" + AXPublicErrorPathKey = "public_error_path" + AXPublicErrorHintKey = "public_error_hint" + AXPublicErrorDetailKey = "public_error_message" ) var AXActionTypes = []string{ diff --git a/computer-use/internal/contract/tool_contract.go b/computer-use/internal/contract/tool_contract.go index 7f7142f..fcacda9 100644 --- a/computer-use/internal/contract/tool_contract.go +++ b/computer-use/internal/contract/tool_contract.go @@ -12,6 +12,11 @@ const ( PublicToolGUI = "GUI" ) +const keyChordDescription = "One chord, e.g. [\"CTRL\",\"S\"]. " + + "PAGEUP/PAGE_UP/PAGEDOWN/PAGE_DOWN. Linux named keys are case-insensitive, " + + "including KP_ADD, KP_SUBTRACT, KP_MULTIPLY, KP_DIVIDE, KP_DECIMAL, KP_ENTER, " + + "KP_0 through KP_9 and mapped X11 names. Use PLUS for '+'." + const ( GUIMaxActions = 64 GUIMaxScrollDelta = 2000 @@ -27,6 +32,7 @@ const ( GUIMaxSettleMS = 3000 AXBatchMaxActions = 64 AXBatchMaxText = 20000 + AXMaxReadText = 4000 MaxClickCount = 10 MaxScrollPages = 100 MaxVerifyTimeout = 60000 @@ -187,7 +193,7 @@ func axInputSchema() map[string]any { "app": nonBlankString(AppArgumentHelp), "window_ref": map[string]any{ "type": "string", "pattern": "^w[1-9][0-9]*$", - "description": "observe/find: window to read. act: omit when actions have refs, which name the window; it only sends ref-less keys/text to wN instead of the focused window.", + "description": "observe/find: window to read. act: omit when actions have refs, which name the window; otherwise sets the initial target for ref-less steps instead of the focused window.", }, "scope": map[string]any{"type": "string", "enum": AXScopeValues, "description": "observe/find; default window."}, "observe": map[string]any{ @@ -198,6 +204,15 @@ func axInputSchema() map[string]any { "type": "string", "pattern": "^(focused|r[1-9][0-9]*)$", "description": "observe: subtree at rN or the focused element.", }, + "read_text": map[string]any{ + "type": "object", "additionalProperties": false, "required": []string{"field"}, + "description": "Linux observe only: read one live root rN without input or a tree. The ref identifies its window; omit app, scope and observe. Offsets count Unicode code points. Follow next_read to continue; pages are live reads, not a frozen document.", + "properties": map[string]any{ + "field": map[string]any{"type": "string", "enum": []string{"name", "value", "description"}, "description": "value reads the node's AT-SPI Text interface; unavailable text returns an error. name and description read those attributes."}, + "start": map[string]any{"type": "integer", "minimum": 0, "maximum": 2147483647, "description": "Start offset, inclusive; default 0."}, + "limit": map[string]any{"type": "integer", "minimum": 1, "maximum": AXMaxReadText, "description": "Maximum code points; default 4000."}, + }, + }, "query": map[string]any{"$ref": "#/$defs/query"}, "limit": map[string]any{"type": "integer", "minimum": 1, "maximum": MaxAXFindPageSize, "description": "find page size; default 20."}, "cursor": map[string]any{ @@ -213,7 +228,7 @@ func axInputSchema() map[string]any { "type": map[string]any{"type": "string", "enum": AXActionTypes}, "ref": map[string]any{ "type": "string", "pattern": "^[rw][1-9][0-9]*$", - "description": "rN element; wN for activate/close. Omit for keys/text to the focused window.", + "description": "rN element; wN for activate/close. Omit for keys/text to the batch target (focused window by default).", }, "to_ref": map[string]any{ "type": "string", "pattern": "^r[1-9][0-9]*$", @@ -227,16 +242,20 @@ func axInputSchema() map[string]any { "keys": map[string]any{ "type": "array", "minItems": 1, "maxItems": GUIMaxKeys, "items": map[string]any{"type": "string", "pattern": "^\\S(.*\\S)?$", "maxLength": GUIMaxKeyLength}, - "description": "One chord, e.g. [\"CTRL\",\"S\"].", + "description": keyChordDescription, }, "value": map[string]any{"type": "string", "maxLength": MaxActionText}, "direction": map[string]any{"type": "string", "enum": []string{"up", "down", "left", "right"}}, "pages": map[string]any{"type": "number", "exclusiveMinimum": 0, "maximum": MaxScrollPages}, - "window": nonBlankString("wait: a top-level window whose title contains this, on the whole desktop."), - "query": map[string]any{"$ref": "#/$defs/query"}, - "state": nonBlankString("wait: ref state, e.g. checked."), - "gone": map[string]any{"type": "boolean", "description": "wait until it is absent."}, - "settle": map[string]any{"type": "boolean", "description": "wait: true waits until the UI changes after the step before it, then stops changing."}, + "allow_global_input": map[string]any{ + "type": "boolean", "default": true, + "description": "Linux click/scroll only. false forbids driver focus/raise and pointer/keyboard synthesis. Semantic actions may themselves change app focus. Unsupported uses reject the batch before input.", + }, + "window": nonBlankString("wait: a top-level window whose title contains this, on the whole desktop."), + "query": map[string]any{"$ref": "#/$defs/query"}, + "state": nonBlankString("wait: ref state, e.g. checked."), + "gone": map[string]any{"type": "boolean", "description": "wait until it is absent."}, + "settle": map[string]any{"type": "boolean", "description": "wait: true waits until the UI changes after the step before it, then stops changing."}, "timeout_ms": map[string]any{ "type": "integer", "minimum": 1, "maximum": MaxVerifyTimeout, "description": "wait: default 5000.", @@ -269,7 +288,7 @@ func guiActionProperty(includeScreenshot bool) map[string]any { } keyArray := map[string]any{ "type": "array", - "description": "PAGEUP/PAGE_UP/PAGEDOWN/PAGE_DOWN.", + "description": keyChordDescription, "minItems": 1, "maxItems": GUIMaxKeys, "items": map[string]any{ @@ -419,12 +438,16 @@ func GUIInputSchema() map[string]any { // axToolDescription is model-visible on every client, including ones that // compact the input schema. Clients may truncate descriptions at 1,000 bytes. -const axToolDescription = "Accessibility for application windows. Modes: " + +const axToolDescription = "Accessibility for application windows. Linux: observe with root rN and read_text reads bounded pages of full name, value (AT-SPI Text), or description; follow next_read. Modes: " + "apps lists confirmed windows (wN, title, app, active) and reports incomplete inventories; missing windows then remain unknown. observe returns the window tree with " + "rN refs. find returns matching rN. act runs ordered steps, checks them before input, " + "stops at the first failed step, and reports each target's change, windows " + - "opened/closed/focused, and a diff. Copy rN/wN exactly; stale refs fail. keypress/type " + - "without ref go to the focused window. select marks ref, its text substring, or ref..to_ref " + + "opened/closed/focused, and a diff. " + + "activate with wN or title switches the batch target for later refs and ref-less steps. " + + "Other-window refs fail until another activate selects their window. " + + "Each step reports its window; final observation uses the last executed window. " + + "Copy rN/wN exactly; stale refs fail. keypress/type " + + "without ref use the batch target, initially the focused window unless refs or window_ref select it. select marks ref, its text substring, or ref..to_ref " + "(paragraphs, cells) and reports the selection read back. activate/close take a title substring or wN. " + "A wait step sends no input and fails the batch on timeout: window (title, any app), " + "query (as find), ref with value/state/text, or text; gone inverts. settle needs no " + diff --git a/computer-use/internal/core/app_snapshot.go b/computer-use/internal/core/app_snapshot.go index 13d2b6a..9c74ad9 100644 --- a/computer-use/internal/core/app_snapshot.go +++ b/computer-use/internal/core/app_snapshot.go @@ -64,6 +64,8 @@ type windowSelection struct { } type ElementRecord struct { + TextTruncated []string `json:"textTruncated,omitempty"` + TextLengths map[string]int `json:"textLengths,omitempty"` Index int `json:"index"` RuntimeID []int `json:"runtimeId,omitempty"` RuntimeIDAliases [][]int `json:"runtimeAliases,omitempty"` @@ -287,6 +289,9 @@ func (s *AppSnapshot) renderedText() string { lines = append(lines, "windows:") lines = append(lines, renderedWindowInventory(s.Windows)...) } + if note := s.contentUnavailableNote(); note != "" { + lines = append(lines, note) + } if normalizedAXScope(s.Scope) == contract.AXScopeContent && s.scopeRoot != nil { lines = append(lines, fmt.Sprintf( "scope: content root=%s %q", diff --git a/computer-use/internal/core/app_snapshot_content.go b/computer-use/internal/core/app_snapshot_content.go new file mode 100644 index 0000000..df23f5e --- /dev/null +++ b/computer-use/internal/core/app_snapshot_content.go @@ -0,0 +1,41 @@ +package core + +import ( + "strings" + + "github.com/microsoft/Hybrid-Computer-Use/computer-use/internal/contract" +) + +func (snapshot *AppSnapshot) contentUnavailableNote() string { + if snapshot == nil || snapshot.WindowAbsent || + !validInternalWindowHandle(snapshot.WindowHandle) || + len(snapshot.Elements) != 1 || snapshot.deliveryOnly || snapshot.lightReadback || + snapshot.FindCandidatesOnly || len(snapshot.FindResume) != 0 || snapshot.findResume != nil || + snapshot.observationRoot != nil || snapshot.projection.Root != "" || + normalizedAXScope(snapshot.Scope) != contract.AXScopeWindow || + normalizedAXScope(snapshot.projection.Scope) != contract.AXScopeWindow { + return "" + } + coverageNote := "" + coverage := providerCoverageSummary(snapshot.Annotations) + if coverage.ProjectionHidden != 0 || coverage.ExpandableHidden != 0 || + coverage.AcquisitionIncomplete != 0 { + coverageNote = "This read has coverage gaps; omitted content may still exist. " + } + root := &snapshot.Elements[0] + if len(root.RuntimeID) > 1 || len(root.TextTruncated) != 0 || + strings.TrimSpace(root.Value) != "" || strings.TrimSpace(root.Description) != "" || + strings.TrimSpace(root.Placeholder) != "" || strings.TrimSpace(snapshot.SelectedText) != "" { + return "" + } + switch ontologyProviderRole(root) { + case "frame", "window", "dialog", "alert": + return "AX content unavailable in this snapshot: only the top-level window root was returned. " + + coverageNote + + "Visible emptiness, renderer crash, and navigation outcome are unknown. " + + "Inspect a screenshot. Verify the outcome of any prior delete or submit before repeating it. " + + "If the app supports reloading and it is safe, reload, then observe again." + default: + return "" + } +} diff --git a/computer-use/internal/core/app_snapshot_content_test.go b/computer-use/internal/core/app_snapshot_content_test.go new file mode 100644 index 0000000..8e2a561 --- /dev/null +++ b/computer-use/internal/core/app_snapshot_content_test.go @@ -0,0 +1,272 @@ +package core + +import ( + "reflect" + "strings" + "testing" + + "github.com/microsoft/Hybrid-Computer-Use/computer-use/internal/contract" +) + +func contentRootSnapshot() *AppSnapshot { + backend := &reportBackend{windows: []reportWindow{reportMain()}} + snapshot := backend.snapshot(reportMain()) + snapshot.Elements = snapshot.Elements[:1] + snapshot.TreeLines = snapshot.TreeLines[:1] + snapshot.SnapshotID = "root-only" + snapshot.projection = snapshotProjection{View: contract.TreeViewWorking, Scope: contract.AXScopeWindow} + return snapshot +} + +func TestAppSnapshotContentDiagnosisRequiresWholeWindowRoot(test *testing.T) { + for _, role := range []string{"frame", "window", "dialog", "alert"} { + test.Run(role, func(test *testing.T) { + snapshot := contentRootSnapshot() + snapshot.Elements[0].ControlType = role + if snapshot.contentUnavailableNote() == "" { + test.Fatal("complete root-only window has no content diagnostic") + } + }) + } + var absent *AppSnapshot + if absent.contentUnavailableNote() != "" { + test.Fatal("nil snapshot was diagnosed") + } + for _, scenario := range []struct { + name string + change func(*AppSnapshot) + }{ + {"static descendant", func(snapshot *AppSnapshot) { + snapshot.Elements = append(snapshot.Elements, ElementRecord{ + Index: 1, RuntimeID: []int{0, 0}, ControlType: "label", Name: "Saved successfully", + }) + }}, + {"anonymous descendant", func(snapshot *AppSnapshot) { + snapshot.Elements = append(snapshot.Elements, ElementRecord{ + Index: 1, RuntimeID: []int{0, 0}, ControlType: "panel", + }) + }}, + {"static root", func(snapshot *AppSnapshot) { snapshot.Elements[0].ControlType = "text" }}, + {"document root", func(snapshot *AppSnapshot) { snapshot.Elements[0].ControlType = "document web" }}, + {"root value", func(snapshot *AppSnapshot) { snapshot.Elements[0].Value = "Saved successfully" }}, + {"root description", func(snapshot *AppSnapshot) { snapshot.Elements[0].Description = "Status details" }}, + {"root placeholder", func(snapshot *AppSnapshot) { snapshot.Elements[0].Placeholder = "Search" }}, + {"selected text", func(snapshot *AppSnapshot) { snapshot.SelectedText = "Selected content" }}, + {"missing root", func(snapshot *AppSnapshot) { snapshot.Elements = nil }}, + {"window absent", func(snapshot *AppSnapshot) { snapshot.WindowAbsent = true }}, + {"no window identity", func(snapshot *AppSnapshot) { snapshot.WindowHandle = "" }}, + {"weak window identity", func(snapshot *AppSnapshot) { snapshot.WindowHandle = "weak:pid10/win0/frame" }}, + {"delivery only", func(snapshot *AppSnapshot) { snapshot.deliveryOnly = true }}, + {"light readback", func(snapshot *AppSnapshot) { snapshot.lightReadback = true }}, + {"find candidates", func(snapshot *AppSnapshot) { snapshot.FindCandidatesOnly = true }}, + {"find continuation", func(snapshot *AppSnapshot) { snapshot.FindResume = []int{0, 1} }}, + {"content scope", func(snapshot *AppSnapshot) { snapshot.Scope = contract.AXScopeContent }}, + {"content projection", func(snapshot *AppSnapshot) { snapshot.projection.Scope = contract.AXScopeContent }}, + {"rooted observation", func(snapshot *AppSnapshot) { snapshot.observationRoot = &snapshot.Elements[0] }}, + {"rooted projection", func(snapshot *AppSnapshot) { snapshot.projection.Root = `{"focused":true}` }}, + {"descendant-only record", func(snapshot *AppSnapshot) { snapshot.Elements[0].RuntimeID = []int{0, 1} }}, + {"truncated root text", func(snapshot *AppSnapshot) { snapshot.Elements[0].TextTruncated = []string{"name"} }}, + } { + test.Run(scenario.name, func(test *testing.T) { + snapshot := contentRootSnapshot() + scenario.change(snapshot) + if note := snapshot.contentUnavailableNote(); note != "" { + test.Fatalf("partial or content-bearing snapshot was diagnosed: %s", note) + } + snapshot.Annotations = []axProviderAnnotation{{Kind: "provider_omission", Count: 10, Reason: "filtered", ParentRuntimeID: []int{0}}} + if note := snapshot.contentUnavailableNote(); note != "" { + test.Fatalf("coverage qualification bypassed the diagnosis exclusion: %s", note) + } + }) + } +} + +func TestAppSnapshotContentDiagnosisQualifiesCoverageGaps(test *testing.T) { + for _, reason := range []string{ + "filtered", "pressure", "collapsed_menu", "provider_children_not_rendered", "modal_dialog", + "time", "chars", "budget", "depth", "render_error", "fanout", "outside_view", + "provider_probe_error", "null_reference", "filtered,depth", "unknown_reason", "", + } { + test.Run(reason, func(test *testing.T) { + snapshot := contentRootSnapshot() + snapshot.Annotations = []axProviderAnnotation{{Kind: "provider_omission", Count: 1, Reason: reason}} + note := snapshot.contentUnavailableNote() + if !strings.Contains(note, "AX content unavailable in this snapshot:") || + !strings.Contains(note, "This read has coverage gaps; omitted content may still exist.") { + test.Fatalf("root-only snapshot lost its qualified diagnosis: %s", note) + } + }) + } +} + +func TestAppSnapshotContentDiagnosisPreservesCoverage(test *testing.T) { + for _, scenario := range []struct { + reason, evidenceKey string + }{ + {"filtered", "projection_hidden_nodes"}, + {"collapsed_menu", "expandable_hidden_nodes"}, + {"provider_probe_error", "coverage_incomplete_nodes"}, + {"null_reference", "coverage_incomplete_nodes"}, + } { + test.Run(scenario.reason, func(test *testing.T) { + service := NewService() + snapshot := contentRootSnapshot() + original := []axProviderAnnotation{{ + Kind: "provider_omission", Count: 10, Reason: scenario.reason, ParentRuntimeID: []int{0}, + }} + snapshot.Annotations = append([]axProviderAnnotation(nil), original...) + if err := service.BuildAndRenderAXSnapshot(snapshot, contract.TreeViewWorking); err != nil { + test.Fatal(err) + } + providerNote := "The accessibility read omitted content." + result := service.getAppStateResult(snapshot, []string{providerNote}, false) + text := ResultText(result) + for _, required := range []string{ + "AX content unavailable in this snapshot: only the top-level window root was returned.", + "This read has coverage gaps; omitted content may still exist.", + "Visible emptiness, renderer crash, and navigation outcome are unknown.", + providerNote, + } { + if !strings.Contains(text, required) { + test.Fatalf("whole-window observation lost %q:\n%s", required, text) + } + } + if result.IsError || result.StructuredContent[scenario.evidenceKey] != 10 || + !reflect.DeepEqual(snapshot.Annotations, original) { + test.Fatalf("diagnosis changed provider coverage or result: %#v annotations=%#v", + result, snapshot.Annotations) + } + if scenario.evidenceKey == "coverage_incomplete_nodes" { + public := PrepareAXEvidence(EnforceAXResponseBudget(WithAXModeEvidence(result, contract.AXModeObserve))) + if public.StructuredContent["coverage"] != "partial" || public.StructuredContent["tree_complete"] != false { + test.Fatalf("incomplete acquisition was presented as complete: %#v", public.StructuredContent) + } + } + }) + } +} + +func TestAppSnapshotContentDiagnosisSurvivesTreeReference(test *testing.T) { + service := NewService() + snapshot := contentRootSnapshot() + if err := service.BuildAndRenderAXSnapshot(snapshot, contract.TreeViewWorking); err != nil { + test.Fatal(err) + } + first := service.getAppStateResult(snapshot, nil, false) + text := ResultText(first) + for _, required := range []string{ + "AX content unavailable in this snapshot:", + "Visible emptiness, renderer crash, and navigation outcome are unknown", + "Inspect a screenshot", "Verify the outcome of any prior delete or submit before repeating it", + "If the app supports reloading and it is safe, reload, then observe again", + } { + if !strings.Contains(text, required) { + test.Fatalf("missing diagnosis or recovery guidance %q:\n%s", required, text) + } + } + if strings.Contains(text, "coverage gaps") || strings.Contains(text, "omitted content may still exist") { + test.Fatalf("complete root-only snapshot acquired a coverage qualifier:\n%s", text) + } + if strings.Index(text, "AX content unavailable") < strings.Index(text, "window:") || + strings.Index(text, "AX content unavailable") > strings.Index(text, "ax:") { + test.Fatalf("diagnosis is not between window identity and tree:\n%s", text) + } + repeated := service.getAppStateResult(snapshot, nil, false) + if repeated.StructuredContent["tree_reference_only"] != true || + !strings.Contains(ResultText(repeated), "AX content unavailable in this snapshot:") { + test.Fatalf("tree reference lost current content diagnosis: %#v\n%s", repeated.StructuredContent, ResultText(repeated)) + } + if first.IsError || repeated.IsError { + test.Fatal("content diagnosis changed successful observation into an error") + } +} + +func TestAppSnapshotContentDiagnosisSurvivesDiff(test *testing.T) { + for _, scenario := range []struct { + name string + withContent bool + wantKind string + coverageReason string + }{ + {"unchanged root", false, "diff", ""}, + {"content disappears", true, "working_tree", ""}, + {"filtered root", false, "diff", "filtered"}, + } { + test.Run(scenario.name, func(test *testing.T) { + service := NewService() + before := contentRootSnapshot() + if scenario.withContent { + backend := &reportBackend{windows: []reportWindow{reportMain()}} + before = backend.snapshot(reportMain()) + before.projection = snapshotProjection{View: contract.TreeViewWorking, Scope: contract.AXScopeWindow} + } + if err := service.BuildAndRenderAXSnapshot(before, contract.TreeViewWorking); err != nil { + test.Fatal(err) + } + service.getAppStateResult(before, nil, false) + after := contentRootSnapshot() + if scenario.coverageReason != "" { + after.Annotations = []axProviderAnnotation{{ + Kind: "provider_omission", Count: 10, Reason: scenario.coverageReason, ParentRuntimeID: []int{0}, + }} + } + if err := service.BuildAndRenderAXSnapshot(after, contract.TreeViewWorking); err != nil { + test.Fatal(err) + } + result := service.renderAXBatchFinalObservation(after, nil, observation{mode: contract.ObserveDiff}) + if result.IsError || result.StructuredContent["final_observation_kind"] != scenario.wantKind || + !strings.Contains(ResultText(result), "AX content unavailable in this snapshot:") { + test.Fatalf("diff observation lost current content diagnosis: %#v\n%s", result.StructuredContent, ResultText(result)) + } + if strings.Contains(ResultText(result), "This read has coverage gaps;") != (scenario.coverageReason != "") { + test.Fatalf("diff changed the coverage qualification: %s", ResultText(result)) + } + }) + } +} + +func TestAppSnapshotContentDiagnosisPreservesActionUncertainty(test *testing.T) { + service := NewService() + backend := &recordingPlatformBackend{accessibilityResponse: &ProviderAccessibilityResult{ + OK: true, Snapshot: contentRootSnapshot(), + }} + service.Backend = backend + window, _ := reportObserve(test, service) + backend.accessibilityResponse = &ProviderAccessibilityResult{ + OK: true, Snapshot: contentRootSnapshot(), + Mutation: &ProviderMutationFacts{InputEventsEmitted: true, DesktopStateMutated: true}, + ActionEvidence: &ProviderActionEvidence{InputEmitted: true, Route: "synthesis"}, + } + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "window_ref": window, "observe": "tree", "actions": keys("Delete"), + }) + if result.IsError { + test.Fatalf("root-only action failed: %s", ResultText(result)) + } + public := WithAXTextEvidence(PrepareAXEvidence(result)) + text := ResultText(public) + if !strings.Contains(text, `"effect_unknown":true`) || + !strings.Contains(text, "AX content unavailable in this snapshot:") { + test.Fatalf("action lost uncertainty or content diagnosis:\n%s", text) + } + if len(backend.actionRequests) != 1 || len(backend.observeRequests) != 1 { + test.Fatalf("diagnosis caused extra actions or observations: actions=%d observations=%d", + len(backend.actionRequests), len(backend.observeRequests)) + } +} + +func TestAppSnapshotContentDiagnosisDoesNotLeakFromLightReadback(test *testing.T) { + service, backend := rootOnlyService(test, false, reportMain()) + window, _ := reportObserve(test, service) + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "window_ref": window, "observe": "tree", "actions": keys("a", "b", "c", "d"), + }) + if result.IsError || strings.Contains(ResultText(result), "AX content unavailable in this snapshot:") { + test.Fatalf("light readback produced a false diagnosis:\n%s", ResultText(result)) + } + if backend.rootOnlyReads != 2 || backend.fullReads != 2 || + len(backend.actionRequests) != 4 || len(backend.observeRequests) != 1 { + test.Fatalf("read or action counts changed: root=%d full=%d actions=%d observations=%d", + backend.rootOnlyReads, backend.fullReads, len(backend.actionRequests), len(backend.observeRequests)) + } +} diff --git a/computer-use/internal/core/ax_act_report.go b/computer-use/internal/core/ax_act_report.go index f6678b9..59da8b6 100644 --- a/computer-use/internal/core/ax_act_report.go +++ b/computer-use/internal/core/ax_act_report.go @@ -15,8 +15,8 @@ import ( // effect: r58 checkbox "Interface" checked=false→true // windows: opened w5 "Insert Sheet" (dialog, focused); closed w3 "Print"; focus w2 // -// One effect line per action that named an element ref, compared between -// the read before that action and the next read after it. The windows line +// One effect line per type or action that named an element ref, compared +// between the reads around that action. The windows line // compares the application's top-level windows before the first action with // those after the last one. Both come from reads the batch makes anyway; the // report adds no provider call. @@ -34,8 +34,10 @@ type actReportTrace struct { observe string // start is the snapshot before any input; before[i] the snapshot just // before action i ran. - start *AppSnapshot - before map[int]*AppSnapshot + start *AppSnapshot + before map[int]*AppSnapshot + afterAction map[int]*AppSnapshot + stepWindows map[int]string } func newActReportTrace(obs observation) *actReportTrace { @@ -61,9 +63,29 @@ func (t *actReportTrace) recordBefore(s *Service, index int) { t.before[index] = s.currentSnapshot(t.appToken) } +func (t *actReportTrace) recordWindow(index int, ref string) { + if t.stepWindows == nil { + t.stepWindows = map[int]string{} + } + t.stepWindows[index] = ref +} + +func (t *actReportTrace) recordAfter(s *Service, index int) { + if t.afterAction == nil { + t.afterAction = map[int]*AppSnapshot{} + } + t.afterAction[index] = s.currentSnapshot(t.appToken) +} + // after returns the first snapshot read after action index, or nil when no // read happened after it. func (t *actReportTrace) after(s *Service, index int) *AppSnapshot { + if after, present := t.afterAction[index]; present { + if after == t.before[index] { + return nil + } + return after + } next := -1 for candidate := range t.before { if candidate > index && (next < 0 || candidate < next) { @@ -98,25 +120,38 @@ func (s *Service) withActReport( } } lines := []string{} - // The effect counts as observed only when every action named an element - // and each of those elements visibly changed; a keypress or a no-change - // action keeps the call's uncertainty. + // The effect counts as observed only when every action has its own + // evidence: a value change for type, a target change for other actions. verified := completed == len(actions) for index, action := range actions { if action.actionType == "wait" { continue } - if !isShortAXRef(action.targetRef) { + if action.actionType != "type" && !isShortAXRef(action.targetRef) { verified = false continue } - if _, ran := trace.before[index]; !ran { - continue - } if index > completed { break } - line, changed := s.actEffectLine(trace, index, action.targetRef) + if _, ran := trace.before[index]; !ran && (action.actionType != "type" || index >= completed) { + continue + } + var line string + var changed bool + if action.actionType == "type" { + switch { + case index > 0 && actions[index-1].actionType != "wait" && + trace.before[index] != nil && trace.before[index] == trace.before[index-1]: + line = axReportEffectPrefix + "type unknown (no separate read before the action)" + case trace.before[index] != nil && index+1 < completed && trace.before[index+1] == nil: + line = axReportEffectPrefix + "type unknown (no separate read after the action)" + default: + line, changed = s.actTypeEffectLine(trace, index, action.targetRef) + } + } else { + line, changed = s.actEffectLine(trace, index, action.targetRef) + } lines = append(lines, line) verified = verified && changed } @@ -131,6 +166,101 @@ func (s *Service) withActReport( return result } +func (s *Service) actTypeEffectLine(trace *actReportTrace, index int, ref string) (string, bool) { + head := axReportEffectPrefix + "type" + beforeSnapshot := trace.before[index] + if beforeSnapshot == nil || beforeSnapshot.AXSnapshot == nil { + return head + " unknown (no read before the action)", false + } + if ref == "" { + var reason string + ref, reason = axReportTypeFocusRef(beforeSnapshot) + if reason != "" { + return head + " unknown (" + reason + ")", false + } + } + head += " " + ref + before := axSnapshotNodeByRef(beforeSnapshot, ref) + if before == nil { + return head + " unknown (the read before the action did not include it)", false + } + head += " " + axReportNodeLabel(before) + afterSnapshot := trace.after(s, index) + if afterSnapshot == nil { + reason := "no read after the action" + if trace.observe == contract.ObserveNone { + reason = `observe "none" skipped the read after the action` + } + return head + " unknown (" + reason + ")", false + } + if afterSnapshot.WindowHandle != beforeSnapshot.WindowHandle { + return head + " unknown (the read after the action is of " + axReportSnapshotWindow(afterSnapshot) + ")", false + } + after := axSnapshotNodeByRef(afterSnapshot, ref) + if after == nil { + return head + " unknown (the read after the action did not include it)", false + } + old, now := axReportTypeValue(before), axReportTypeValue(after) + if old == "" { + line := head + " unknown (value before the action is empty or unavailable)" + if now != "" { + line += "; value after=" + strconv.Quote(truncateRunes(now, 60)) + } + return line, false + } + if now == "" { + return head + " unknown (value after the action is empty or unavailable)", false + } + if old == now { + return head + " no visible change (value unchanged; text input not verified)", false + } + from, to := strconv.Quote(truncateRunes(old, 60)), strconv.Quote(truncateRunes(now, 60)) + if from == to { + return head + " value changed (difference beyond the displayed prefix)", true + } + return head + " value=" + from + "→" + to, true +} + +func axReportTypeFocusRef(snapshot *AppSnapshot) (string, string) { + if !snapshot.AXSnapshot.RefInventoryComplete { + return "", "the read before the action is incomplete; focus uniqueness is unknown" + } + roots := snapshot.AXSnapshot.CanonicalRoots + if len(roots) == 0 { + roots = snapshot.AXSnapshot.Roots + } + var focused *AXNode + count := 0 + var visit func([]*AXNode) + visit = func(nodes []*AXNode) { + for _, node := range nodes { + if node == nil { + continue + } + if node.States.Active == TriTrue { + focused = node + count++ + } + visit(node.Children) + } + } + visit(roots) + if count != 1 || focused.Role != "textbox" || !focused.Capabilities.EditText { + return "", "no unique focused editable textbox in the read before the action" + } + if !isShortAXRef(focused.Ref) { + return "", "the focused editable textbox has no ref" + } + return focused.Ref, "" +} + +func axReportTypeValue(node *AXNode) string { + if node.SourceRecord != nil { + return node.SourceRecord.Value + } + return node.Properties.Value +} + // actEffectLine compares one element between the reads around its action. // changed reports a visible change of the element. func (s *Service) actEffectLine(trace *actReportTrace, index int, ref string) (string, bool) { @@ -498,6 +628,27 @@ func axReportWindows(snapshot *AppSnapshot) []axReportWindow { // actWindowsLine compares the application's windows before the first action // with those after the last one. func (s *Service) actWindowsLine(trace *actReportTrace, result ToolCallResult) string { + windows := map[string]bool{} + indices := make([]int, 0, len(trace.stepWindows)) + for index, ref := range trace.stepWindows { + windows[ref] = true + indices = append(indices, index) + } + if len(windows) > 1 { + sort.Ints(indices) + steps := make([]string, 0, len(indices)) + for _, index := range indices { + steps = append(steps, strconv.Itoa(index+1)+"="+trace.stepWindows[index]) + } + final := trace.windowRef + if ref, ok := result.StructuredContent["final_observation_window_ref"].(string); ok && ref != "" { + final = ref + } else if snapshot := s.currentSnapshot(trace.appToken); snapshot != nil && snapshot.publicWindowRef != "" && + result.StructuredContent["final_observation_returned"] == true { + final = snapshot.publicWindowRef + } + return axReportWindowsPrefix + "step targets " + strings.Join(steps, ", ") + "; final " + final + } closedBound := result.StructuredContent["target_window_absent"] == true || result.StructuredContent["close_effect_verified"] == true final := s.currentSnapshot(trace.appToken) diff --git a/computer-use/internal/core/ax_act_report_test.go b/computer-use/internal/core/ax_act_report_test.go index 6e323d8..69972ec 100644 --- a/computer-use/internal/core/ax_act_report_test.go +++ b/computer-use/internal/core/ax_act_report_test.go @@ -26,9 +26,10 @@ type reportWindow struct { // reads the window back, as the real provider does. type reportBackend struct { recordingPlatformBackend - windows []reportWindow - checked bool - value string + windows []reportWindow + checked bool + value string + textStates string // onAction mutates the state when input arrives. onAction func(b *reportBackend) // skipReadback makes an action return no snapshot (observe "none"). @@ -47,6 +48,7 @@ func (b *reportBackend) snapshot(selected reportWindow) *ProviderSnapshot { checked = "[checked]" } setValue := true + textStates := FirstNonEmpty(b.textStates, "[editable focusable]") snapshot := &AppSnapshot{ App: AppDescriptor{Name: "Editor", PID: 10, ProviderBusName: ":1.84", ProcessGeneration: "pid:10:start:100", InputCapable: true}, @@ -55,14 +57,14 @@ func (b *reportBackend) snapshot(selected reportWindow) *ProviderSnapshot { TreeLines: []string{ "\t0 " + selected.role + " \"" + selected.title + "\"", "\t\t1 check box \"Interface\" " + checked, - "\t\t2 entry \"Size\" [editable]", + "\t\t2 entry \"Size\" " + textStates, "\t\t3 push button \"OK\" [click]", }, Elements: []ElementRecord{ {Index: 0, RuntimeID: []int{0}, ControlType: selected.role, Name: selected.title}, {Index: 1, RuntimeID: []int{0, 0}, ControlType: "check box", Name: "Interface", States: checked}, {Index: 2, RuntimeID: []int{0, 1}, ControlType: "entry", Name: "Size", Value: b.value, - States: "[editable focusable]", SetValueSupported: &setValue}, + States: textStates, SetValueSupported: &setValue}, {Index: 3, RuntimeID: []int{0, 2}, ControlType: "push button", Name: "OK", States: "[has-click-action]"}, {Index: 4, RuntimeID: []int{0, 3}, ControlType: "label", Name: FirstNonEmpty(b.page, "General")}, {Index: 5, RuntimeID: []int{0, 4}, ControlType: "menu item", Name: "Edit", States: "[has-click-action]"}, @@ -274,6 +276,220 @@ func TestActReportShowsAValueChange(t *testing.T) { } } +func TestActReportClickThenRefLessType(t *testing.T) { + for _, outcome := range []string{"unchanged", "changed"} { + t.Run(outcome, func(t *testing.T) { + changed := outcome == "changed" + service, backend := reportService(t, reportMain()) + backend.value = "1" + _, refs := reportObserve(t, service, "Size") + backend.onAction = func(backend *reportBackend) { + backend.textStates = "[editable focusable focused]" + backend.value = "12" + if len(backend.actionRequests) == 2 && changed { + backend.value = "123" + } + } + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "actions": []any{ + map[string]any{"type": "click", "ref": refs["Size"]}, + map[string]any{"type": "type", "text": "3"}, + }, + }) + text := ResultText(result) + if result.IsError || len(backend.actionRequests) != 2 { + t.Fatalf("expected both actions to run: %s", text) + } + if !strings.Contains(text, `effect: `+refs["Size"]+` textbox "Size" active=false→true`) { + t.Fatalf("click focus change missing: %s", text) + } + want := "no visible change (value unchanged; text input not verified)" + if changed { + want = `value="12"→"123"` + } + if !strings.Contains(text, `effect: type `+refs["Size"]+` textbox "Size" `+want) { + t.Fatalf("type must compare against the read after click: %s", text) + } + assertTypeReportEvidence(t, result, changed) + }) + } +} + +func reportTypeSnapshot(value string) *AppSnapshot { + record := &ElementRecord{ControlType: "entry", Name: "Input", Value: value, States: "[editable focused]"} + node := &AXNode{ + Ref: "r1", Role: "textbox", Name: "Input", SourceRecord: record, + States: typedAXStates(record), Capabilities: typedAXCapabilities(record, "textbox"), + Properties: AXProperties{Value: normalizedTreeText(value)}, + } + return &AppSnapshot{WindowHandle: reportMainHandle, WindowTitle: "Report", AXSnapshot: &AXSnapshot{ + CanonicalRoots: []*AXNode{node}, Roots: []*AXNode{node}, RefInventoryComplete: true, + }} +} + +func assertTypeReportEvidence(t *testing.T, result ToolCallResult, observed bool) { + t.Helper() + if got := result.StructuredContent["target_effect_observed"] == true; got != observed { + t.Fatalf("target_effect_observed=%v want %v: %s", got, observed, ResultText(result)) + } + public := ResultText(WithAXTextEvidence(PrepareAXEvidence(result))) + if strings.Contains(public, `"effect_unknown":true`) == observed { + t.Fatalf("public uncertainty disagrees with value evidence: %s", public) + } +} + +func TestActReportTypeRequiresValueEvidence(t *testing.T) { + for _, ref := range []string{"", "r1"} { + for _, test := range []struct { + name, before, after, want string + observed bool + }{ + {"changed", "old", "new", `value="old"→"new"`, true}, + {"unchanged", "old", "old", "no visible change (value unchanged; text input not verified)", false}, + {"empty before", "", "new", `unknown (value before the action is empty or unavailable); value after="new"`, false}, + {"empty after", "old", "", "unknown (value after the action is empty or unavailable)", false}, + {"both unavailable", "", "", "unknown (value before the action is empty or unavailable)", false}, + {"whitespace", "old text", "old text", `value="old text"→"old text"`, true}, + {"newline", "old text", "old\ntext", `value="old text"→"old\ntext"`, true}, + {"clipped suffix", strings.Repeat("a", 80) + "b", strings.Repeat("a", 80) + "c", "value changed", true}, + } { + t.Run(ref+"/"+test.name, func(t *testing.T) { + service := NewService() + before, after := reportTypeSnapshot(test.before), reportTypeSnapshot(test.after) + after.AXSnapshot.Roots[0].States.Active = TriFalse + after.AXSnapshot.Roots[0].States.Selected = TriTrue + service.snapshots["editor"] = after + trace := &actReportTrace{appToken: "editor", start: before, before: map[int]*AppSnapshot{0: before}} + result := service.withActReport(ToolCallResult{StructuredContent: map[string]any{ + "input_events_emitted": true, "effect_unknown": true, + }}, []axBatchAction{{actionType: "type", targetRef: ref, text: "text"}}, trace) + assertTypeReportEvidence(t, result, test.observed) + if text := ResultText(result); !strings.Contains(text, `effect: type r1 textbox "Input" `+test.want) { + t.Fatalf("type report lacks value evidence: %s", text) + } + }) + } + } +} + +func TestActReportTypeMissingEvidence(t *testing.T) { + for _, ref := range []string{"", "r1"} { + for _, scenario := range []string{"no before", "no after", "same snapshot", "observe none", "target gone", "other window", "no value source"} { + t.Run(ref+"/"+scenario, func(t *testing.T) { + service := NewService() + before, after := reportTypeSnapshot("old"), reportTypeSnapshot("new") + trace := &actReportTrace{appToken: "editor", start: before, before: map[int]*AppSnapshot{0: before}} + reason := "" + switch scenario { + case "no before": + trace.before[0] = nil + reason = "no read before the action" + case "no after": + after = nil + reason = "no read after the action" + case "same snapshot": + after = before + reason = "no read after the action" + case "observe none": + after = before + trace.observe = contract.ObserveNone + reason = `observe "none" skipped the read after the action` + case "target gone": + after.AXSnapshot.CanonicalRoots, after.AXSnapshot.Roots = nil, nil + reason = "the read after the action did not include it" + case "other window": + after.WindowHandle = reportDialogHandle + reason = "the read after the action is of" + case "no value source": + after.AXSnapshot.Roots[0].SourceRecord = nil + after.AXSnapshot.Roots[0].Properties.Value = "" + reason = "value after the action is empty or unavailable" + } + service.snapshots["editor"] = after + result := service.withActReport(ToolCallResult{StructuredContent: map[string]any{ + "input_events_emitted": true, "effect_unknown": true, + }}, []axBatchAction{{actionType: "type", targetRef: ref, text: "new"}}, trace) + text := ResultText(result) + if !strings.HasPrefix(text, "effect: type ") || !strings.Contains(strings.Split(text, "\n")[0], "unknown ("+reason) { + t.Fatalf("missing reason %q: %s", reason, text) + } + assertTypeReportEvidence(t, result, false) + }) + } + } +} + +func TestActReportRefLessTypeRequiresUniqueFocusedTextbox(t *testing.T) { + for _, scenario := range []string{"unfocused", "not editable", "not textbox", "two focused", "focused button too", "no ref", "incomplete", "projected roots"} { + t.Run(scenario, func(t *testing.T) { + service := NewService() + before, after := reportTypeSnapshot("old"), reportTypeSnapshot("new") + node := before.AXSnapshot.Roots[0] + reason := "no unique focused editable textbox in the read before the action" + observed := false + switch scenario { + case "unfocused": + node.States.Active = TriUnknown + case "not editable": + node.Capabilities.EditText = false + case "not textbox": + node.Role = "button" + case "two focused", "focused button too": + other := *node + other.Ref = "r2" + if scenario == "focused button too" { + other.Role = "button" + } + node.Children = []*AXNode{&other} + case "no ref": + node.Ref = "" + reason = "the focused editable textbox has no ref" + case "incomplete": + before.AXSnapshot.RefInventoryComplete = false + reason = "the read before the action is incomplete; focus uniqueness is unknown" + case "projected roots": + before.AXSnapshot.CanonicalRoots = nil + observed = true + } + service.snapshots["editor"] = after + trace := &actReportTrace{appToken: "editor", start: before, before: map[int]*AppSnapshot{0: before}} + result := service.withActReport(ToolCallResult{StructuredContent: map[string]any{ + "input_events_emitted": true, "effect_unknown": true, + }}, []axBatchAction{{actionType: "type", text: "new"}}, trace) + want := "unknown (" + reason + ")" + if observed { + want = `value="old"→"new"` + } + if text := ResultText(result); !strings.Contains(strings.Split(text, "\n")[0], want) { + t.Fatalf("type report lacks %q: %s", want, text) + } + assertTypeReportEvidence(t, result, observed) + }) + } +} + +func TestActReportTypeDoesNotShareBatchValueEvidence(t *testing.T) { + for _, shared := range []string{"no intermediate snapshot", "reused intermediate snapshot"} { + t.Run(shared, func(t *testing.T) { + service := NewService() + before, after := reportTypeSnapshot("old"), reportTypeSnapshot("oldnew") + service.snapshots["editor"] = after + trace := &actReportTrace{appToken: "editor", start: before, before: map[int]*AppSnapshot{0: before}} + if shared == "reused intermediate snapshot" { + trace.before[1] = before + } + result := service.withActReport(ToolCallResult{StructuredContent: map[string]any{ + "input_events_emitted": true, "effect_unknown": true, + }}, []axBatchAction{{actionType: "type", text: "new"}, {actionType: "type", text: "ignored"}}, trace) + text := ResultText(result) + if strings.Count(text, "effect: type") != 2 || strings.Count(text, " unknown (") != 2 { + t.Fatalf("both types need separate before/after evidence: %s", text) + } + assertTypeReportEvidence(t, result, false) + }) + } +} + // An action whose element did not change says so; the evidence line then // keeps effect_unknown, because nothing confirms the effect. func TestActReportSaysNoVisibleChange(t *testing.T) { diff --git a/computer-use/internal/core/ax_action_evidence.go b/computer-use/internal/core/ax_action_evidence.go index 62167cd..18d635d 100644 --- a/computer-use/internal/core/ax_action_evidence.go +++ b/computer-use/internal/core/ax_action_evidence.go @@ -328,7 +328,8 @@ func observeNoneNotes(notes []string) []string { if phase == "immediate" && category != "" { continue } - if strings.HasPrefix(note, "[expect]") || + if strings.HasPrefix(note, "[a11y][synthesis]") || + strings.HasPrefix(note, "[expect]") || strings.HasPrefix(note, "[semantic-resolution]") || strings.HasPrefix(note, "[canonical-address]") || strings.Contains(note, `"predicate_verified":`) || @@ -404,6 +405,10 @@ func modelFacingAXNotes(notes []string) []string { } continue } + if strings.HasPrefix(note, "[a11y][synthesis]") { + appendOnce(note) + continue + } if strings.HasPrefix(note, "[a11y]") { if offset := strings.Index(note, "[state]"); offset >= 0 { fact := strings.TrimSpace(note[offset:]) diff --git a/computer-use/internal/core/ax_action_evidence_test.go b/computer-use/internal/core/ax_action_evidence_test.go index 49b2825..cbda391 100644 --- a/computer-use/internal/core/ax_action_evidence_test.go +++ b/computer-use/internal/core/ax_action_evidence_test.go @@ -65,6 +65,22 @@ func TestUnchangedNotesReportOnlyObservedChange(t *testing.T) { } } +func TestAXSynthesisNotesSurviveNormalization(t *testing.T) { + note := "[a11y][synthesis] Synthesized wheel input. Global input may change focus; verify the effect." + for _, notes := range [][]string{ + modelFacingAXNotes([]string{note, note}), + modelFacingAXNotes(observeNoneNotes([]string{note})), + } { + if len(notes) != 1 || notes[0] != note { + t.Fatalf("synthesis warning lost or rewritten: %#v", notes) + } + } + result := compactAXActionContent(TextResult("Note: "+note, false)) + if ResultText(result) != "Note: "+note { + t.Fatalf("batch compaction lost the warning: %s", ResultText(result)) + } +} + func TestFindOmitsInternalContinuityDiagnostics(t *testing.T) { notes := compactFindNotes([]string{ "SNAPSHOT UNCHANGED: a long explanation that should not be repeated.", diff --git a/computer-use/internal/core/ax_action_service.go b/computer-use/internal/core/ax_action_service.go index b87ab1f..3bff9c2 100644 --- a/computer-use/internal/core/ax_action_service.go +++ b/computer-use/internal/core/ax_action_service.go @@ -368,6 +368,10 @@ func applyActionElementResolution( } func (s *Service) click(app, elementIndex, declared string, x, y *float64, clickCount int, mouseButton, clickMethod string, obs observation) ToolCallResult { + return s.clickWithGlobalInput(app, elementIndex, declared, x, y, clickCount, mouseButton, clickMethod, obs, nil) +} + +func (s *Service) clickWithGlobalInput(app, elementIndex, declared string, x, y *float64, clickCount int, mouseButton, clickMethod string, obs observation, allowGlobalInput *bool) ToolCallResult { if app == "" { return TextResult("Missing required argument: app", true) } @@ -402,7 +406,8 @@ func (s *Service) click(app, elementIndex, declared string, x, y *float64, click request := AXServiceRequest{ App: app, Action: AXClickCommand{ - X: x, Y: y, ClickCount: clickCount, + AllowGlobalInput: allowGlobalInput, + X: x, Y: y, ClickCount: clickCount, MouseButton: mouseButton, ClickMethod: clickMethod, }, @@ -448,6 +453,16 @@ func (s *Service) scroll( expectedWindow *ProviderWindowExpectation, pages float64, obs observation, +) ToolCallResult { + return s.scrollWithGlobalInput(app, direction, elementIndex, declared, targetWindowHandle, expectedWindow, pages, obs, nil) +} + +func (s *Service) scrollWithGlobalInput( + app, direction, elementIndex, declared, targetWindowHandle string, + expectedWindow *ProviderWindowExpectation, + pages float64, + obs observation, + allowGlobalInput *bool, ) ToolCallResult { if app == "" { return TextResult("Missing required argument: app", true) @@ -470,7 +485,8 @@ func (s *Service) scroll( return s.actionResult(app, AXServiceRequest{ App: app, Action: axScrollCommand{ - Direction: normalized, Pages: pages, WindowCentre: true, + AllowGlobalInput: allowGlobalInput, + Direction: normalized, Pages: pages, WindowCentre: true, }, TargetWindowHandle: targetWindowHandle, ExpectedWindow: expectedWindow, @@ -485,7 +501,7 @@ func (s *Service) scroll( } request := AXServiceRequest{ App: app, - Action: axScrollCommand{Direction: normalized, Pages: pages}, + Action: axScrollCommand{Direction: normalized, Pages: pages, AllowGlobalInput: allowGlobalInput}, TargetWindowHandle: targetWindowHandle, IncludeScreenshot: obs.resolvedScreenshot(), Observe: obs.resolvedMode(), CaptureSnapshot: obs.captureSnapshot, LightReadback: obs.lightReadback, diff --git a/computer-use/internal/core/ax_argument_hint_test.go b/computer-use/internal/core/ax_argument_hint_test.go index e2a73f6..2dac3a5 100644 --- a/computer-use/internal/core/ax_argument_hint_test.go +++ b/computer-use/internal/core/ax_argument_hint_test.go @@ -14,6 +14,17 @@ import ( // TestEveryAXArgumentErrorHintDecodes checks that the table reaches every // axInvalid call in ax_public_request.go. var axBadCalls = []string{ + `{"mode":"observe","read_text":{"field":"value"}}`, + `{"mode":"observe","root":"r1","read_text":{"field":"value"},"scope":"window"}`, + `{"mode":"observe","root":"r1","read_text":{"field":"value"},"app":"Editor"}`, + `{"mode":"observe","root":"r1","read_text":{"field":"value"},"observe":"tree"}`, + `{"mode":"observe","root":"r1","read_text":null}`, + `{"mode":"observe","root":"r1","read_text":{}}`, + `{"mode":"observe","root":"r1","read_text":{"field":"bad"}}`, + `{"mode":"observe","root":"r1","read_text":{"field":"value","start":-1}}`, + `{"mode":"observe","root":"r1","read_text":{"field":"value","limit":4001}}`, + `{"mode":"observe","root":"r1","read_text":{"field":"value","limit":true}}`, + `{"mode":"observe","root":"r1","read_text":{"field":"value","extra":true}}`, // mode `{}`, `{"mode":"assert_ax"}`, @@ -77,6 +88,7 @@ var axBadCalls = []string{ `{"mode":"act","actions":[{"type":"activate","ref":"w3","window_ref":"w3"}]}`, `{"mode":"act","actions":[{"type":"click","ref":"r??"}]}`, `{"mode":"act","actions":[{"type":"click","ref":"w1"}]}`, + `{"mode":"act","actions":[{"type":"click","ref":"r1","allow_global_input":"false"}]}`, `{"mode":"act","actions":[{"type":"close","ref":"r1"}]}`, `{"mode":"act","actions":[{"type":"click"}]}`, `{"mode":"act","actions":[{"type":"set_value","ref":"r1"}]}`, diff --git a/computer-use/internal/core/ax_batch.go b/computer-use/internal/core/ax_batch.go index e108edc..a73fdfc 100644 --- a/computer-use/internal/core/ax_batch.go +++ b/computer-use/internal/core/ax_batch.go @@ -9,8 +9,9 @@ import ( ) type axBatchAction struct { - actionType string - targetRef string + actionType string + targetRef string + allowGlobalInput *bool // toRef is the range end of a select action. toRef string direction string @@ -31,6 +32,9 @@ func (s *Service) executeAXBatchAction( const noDeclaredTarget = "" switch action.actionType { case "activate", "close": + if window := s.noTreeWindowEntry(action.targetRef); window != nil { + return s.axNoTreeBatchAction(action.actionType, window) + } return s.windowAction( context.AppToken, action.targetRef, @@ -39,7 +43,7 @@ func (s *Service) executeAXBatchAction( obs, ).withEvidenceScope("delivery_or_local_ui") case "click": - return s.click( + return s.clickWithGlobalInput( context.AppToken, action.targetRef, noDeclaredTarget, @@ -49,6 +53,7 @@ func (s *Service) executeAXBatchAction( "left", "auto", obs, + action.allowGlobalInput, ).withEvidenceScope("delivery_or_local_ui") case "keypress": return s.pressKey( @@ -61,7 +66,7 @@ func (s *Service) executeAXBatchAction( obs, ).withEvidenceScope("delivery_or_local_ui") case "scroll": - return s.scroll( + return s.scrollWithGlobalInput( context.AppToken, action.direction, action.targetRef, @@ -70,6 +75,7 @@ func (s *Service) executeAXBatchAction( context.expectedWindow, action.pages, obs, + action.allowGlobalInput, ).withEvidenceScope("delivery_or_local_ui") case "set_value": return s.setValue( @@ -106,15 +112,22 @@ func (s *Service) executeAXBatchAction( } } -// axBatchDecoded runs decoded actions in one window. Window titles and -// the focused window are resolved here, once, before any input; after that -// the batch uses only the resolved wN. func (s *Service) axBatchDecoded( actions []axBatchAction, obs observation, windowRef string, ) ToolCallResult { requested := len(actions) + if s.platformBackend().HostInfo().Platform != "linux" { + for index, action := range actions { + if action.allowGlobalInput != nil { + return axArgErrorResult(axInvalid( + fmt.Sprintf("actions[%d].allow_global_input", index), + "allow_global_input is supported only for Linux AX click and scroll", "", + ), requested) + } + } + } if axBatchOnlyWaits(actions) { return s.axWaitOnly(actions, obs, windowRef) } @@ -133,11 +146,8 @@ func (s *Service) axBatchDecoded( "title %s -> %s %s (%s)", strconv.Quote(actions[index].title), window.ref, strconv.Quote(window.title), window.appName)) } - if window := s.axNoTreeBatchWindow(actions, windowRef); window != nil { - return s.axNoTreeWindowBatch(actions, window, bindingNotes, requested) - } var focus *axDesktopWindow - if windowRef == "" && !axBatchHasRef(actions) { + if windowRef == "" && !axBatchHasRef(actions[:axBatchWindowEnd(actions, 0)]) { var argErr *axArgError focus, argErr = s.resolveFocusedWindow() if argErr != nil { @@ -189,7 +199,17 @@ func (s *Service) axBatchRun( focus *axDesktopWindow, requested int, trace *actReportTrace, -) ToolCallResult { +) (batchResult ToolCallResult) { + var synthesisNotes []string + defer func() { + text := ResultText(batchResult) + for _, note := range synthesisNotes { + if !strings.Contains(text, note) { + batchResult.Content = append(batchResult.Content, ContentItem{Type: "text", Text: note}) + text += "\n" + note + } + } + }() if err := s.preflightPlatformAXKeys(actions); err != nil { return axBatchPreflightError(err, requested) } @@ -198,10 +218,11 @@ func (s *Service) axBatchRun( return axBatchPreflightError(err, requested) } } - context, err := s.deriveAXBatchContext(windowRef, actions, focus) + contexts, err := s.deriveAXBatchContexts(windowRef, actions, focus) if err != nil { return axBatchPreflightError(err, requested) } + context := contexts[0] context.report = trace trace.bind(s, context) lastInput := -1 @@ -210,12 +231,6 @@ func (s *Service) axBatchRun( lastInput = index continue } - if action.wait.ref == "" { - continue - } - if err := s.preflightAXWaitRef(context, action.wait.ref); err != nil { - return axBatchPreflightError(fmt.Errorf("actions[%d].ref: %w", index, err), requested) - } } executionPlan, err := compileAXBatchExecutionPlan(context, actions, obs) @@ -227,7 +242,7 @@ func (s *Service) axBatchRun( // provider path runs keys back to back. guarded := context.boundByFocus || lastInput < len(actions)-1 for _, action := range actions { - guarded = guarded || action.actionType == "wait" + guarded = guarded || action.actionType == "wait" || action.actionType == "activate" } if _, available := orderedInputBackend(s.platformBackend()); available && !guarded { requests, err := providerOrderedInputRequestsFromPlan(context, executionPlan, actions) @@ -245,16 +260,29 @@ func (s *Service) axBatchRun( waitNotes := []string{} for index := 0; index < len(actions); index++ { action := actions[index] + if action.actionType == "activate" { + pending := context.windowsQueuedInputPending + context = contexts[index] + context.windowsQueuedInputPending = pending + context.report = trace + trace.appToken = context.AppToken + trace.windowRef = context.WindowRef + } if action.actionType == "wait" { // A wait step between actions: it sends no input. trace.recordBefore(s, index) + trace.recordWindow(index, context.WindowRef) waitResult := s.runAXWait(context, action.wait) - actionEvidence = append(actionEvidence, axWaitEvidenceEntry(index, waitResult)) + waitEntry := axWaitEvidenceEntry(index, waitResult) + waitEntry["window_ref"] = context.WindowRef + actionEvidence = append(actionEvidence, waitEntry) if !axPostconditionPassed(waitResult) { return s.stopAXBatchAtWait(context, actions, index, waitResult, actionEvidence, waitNotes, obs) } waitNotes = append(waitNotes, firstLine(ResultText(waitResult))) s.rebindAfterWindowWait(context, action.wait, waitResult) + waitEntry["window_ref"] = context.WindowRef + trace.recordWindow(index, context.WindowRef) if index < len(actions)-1 && actions[index+1].actionType != "wait" { if stopped, changed := s.axBatchContinuityStop(context, actions, index, actionEvidence, obs); changed { return stopped @@ -283,6 +311,7 @@ func (s *Service) axBatchRun( context.settleBaseline = s.readAXSettleState(context, nil) } trace.recordBefore(s, index) + trace.recordWindow(index, context.WindowRef) result := ToolCallResult{} actionEntry := map[string]any{} request, orderedRun := orderedRuns[index] @@ -305,7 +334,9 @@ func (s *Service) axBatchRun( s.renderAXBatchFinalObservation(execution.snapshot, execution.notes, obs)) } } else { - if context.windowsQueuedInputPending && action.actionType == "type" && action.text != "" { + if err := s.validateAXActionWindow(context, action, index); err != nil { + result = axBatchPreflightError(err, requested) + } else if context.windowsQueuedInputPending && action.actionType == "type" && action.text != "" { result = TextResult("Earlier queued input has not been confirmed processed; no text was sent. Observe and confirm focus and selection before sending text separately; do not repeat the earlier input.", true) result.StructuredContent = map[string]any{ "failure_code": "desktop_input_failed", "failure_phase": "preparation", @@ -325,9 +356,9 @@ func (s *Service) axBatchRun( context.windowsQueuedInputPending = true } } - if !result.IsError || + if s.noTreeWindowEntry(context.WindowRef) == nil && (!result.IsError || (action.actionType == "close" && - axEvidenceReportsInputEmission(result.StructuredContent)) { + axEvidenceReportsInputEmission(result.StructuredContent))) { result = s.recoverClosedWindowActionObservation( context, action, @@ -336,6 +367,8 @@ func (s *Service) axBatchRun( ) } actionEntry = batchActionEvidenceEntry(index, action.actionType, result) + actionEntry["window_ref"] = context.WindowRef + trace.recordAfter(s, index) context.settleBaseline.notePixelChange(result) } if result.IsError && index+1 < len(actions) && actions[index+1].actionType == "wait" && @@ -347,7 +380,15 @@ func (s *Service) axBatchRun( result.IsError = false actionEntry["call_error"] = false } + for _, line := range strings.Split(ResultText(result), "\n") { + if note := strings.TrimSpace(line); strings.HasPrefix(note, "Note: [a11y][synthesis]") { + synthesisNotes = append(synthesisNotes, note) + } + } if result.IsError { + if axPublicErrorCode(result.StructuredContent) == contract.AXCodeGlobalInputForbidden { + result.StructuredContent[contract.AXPublicErrorCodeKey] = contract.AXCodeGlobalInputForbidden + } actionEvidence = append(actionEvidence, actionEntry) anyInputEmitted, anyPartialPossible, anyMutationUnknown := batchMutationSummary( @@ -369,6 +410,12 @@ func (s *Service) axBatchRun( if result.StructuredContent == nil { result.StructuredContent = map[string]any{} } + if hint := axBatchFailureHint(context, result.StructuredContent); hint != "" { + result.StructuredContent[contract.AXPublicErrorHintKey] = hint + if continuation, ok := result.StructuredContent["continuation"].(map[string]any); ok && hint == GUIScreenshotCall { + continuation["route"] = "GUI" + } + } finalObservationReturned, _ := result.StructuredContent["final_observation_returned"].(bool) preparationFailed := result.StructuredContent["focus_preparation"] != nil || @@ -433,13 +480,18 @@ func (s *Service) axBatchRun( // which then shows the state they waited for. for next := index + 1; next < len(actions); next++ { trace.recordBefore(s, next) + trace.recordWindow(next, context.WindowRef) waitResult := s.runAXWait(context, actions[next].wait) - actionEvidence = append(actionEvidence, axWaitEvidenceEntry(next, waitResult)) + waitEntry := axWaitEvidenceEntry(next, waitResult) + waitEntry["window_ref"] = context.WindowRef + actionEvidence = append(actionEvidence, waitEntry) if !axPostconditionPassed(waitResult) { return s.stopAXBatchAtWait(context, actions, next, waitResult, actionEvidence, waitNotes, obs) } waitNotes = append(waitNotes, firstLine(ResultText(waitResult))) s.rebindAfterWindowWait(context, actions[next].wait, waitResult) + waitEntry["window_ref"] = context.WindowRef + trace.recordWindow(next, context.WindowRef) } if index < len(actions)-1 && obs.resolvedMode() != contract.ObserveNone && action.actionType != "close" { @@ -473,6 +525,14 @@ func (s *Service) axBatchRun( if value, ok := result.StructuredContent["final_observation_returned"].(bool); ok { finalObservationReturned = value } + if finalObservationReturned && contexts[0] != contexts[len(contexts)-1] { + if _, present := result.StructuredContent["final_observation_window_ref"]; !present { + if snapshot := s.currentSnapshot(context.AppToken); snapshot != nil && + snapshot.publicWindowRef != "" { + result.StructuredContent["final_observation_window_ref"] = snapshot.publicWindowRef + } + } + } observationRequired := !finalObservationReturned && obs.resolvedMode() != contract.ObserveNone if value, present := diff --git a/computer-use/internal/core/ax_batch_activation_test.go b/computer-use/internal/core/ax_batch_activation_test.go new file mode 100644 index 0000000..3a4f5fb --- /dev/null +++ b/computer-use/internal/core/ax_batch_activation_test.go @@ -0,0 +1,542 @@ +package core + +import ( + "fmt" + "regexp" + "slices" + "strings" + "testing" + + "github.com/microsoft/Hybrid-Computer-Use/computer-use/internal/contract" +) + +type axActivationBackend struct { + recordingPlatformBackend + windows []reportWindow + owners map[string]AppDescriptor + windowRequests []ProviderWindowActionRequest + deliveries []string + failActivate string + transitionAfterInput string + activationModal *reportWindow + inputModal *reportWindow +} + +func (backend *axActivationBackend) snapshot(handle string) *ProviderSnapshot { + owner := backend.owners[handle] + fixture := &reportBackend{value: "12"} + var selected reportWindow + for _, window := range backend.windows { + if backend.owners[window.handle].Name == owner.Name { + fixture.windows = append(fixture.windows, window) + } + if window.handle == handle { + selected = window + } + } + snapshot := fixture.snapshot(selected) + snapshot.App = owner + for index := range snapshot.Elements { + element := &snapshot.Elements[index] + element.ProviderBusName = owner.ProviderBusName + element.ProcessGeneration = owner.ProcessGeneration + element.ProviderWindowBusName = owner.ProviderBusName + element.ProviderWindowProcessGeneration = owner.ProcessGeneration + } + for index := range snapshot.Windows { + window := &snapshot.Windows[index] + window.ProviderBusName = owner.ProviderBusName + window.ProviderProcessGeneration = owner.ProcessGeneration + } + return snapshot +} + +func (backend *axActivationBackend) ObserveApps(ProviderObserveAppsRequest) (ProviderAccessibilityResult, error) { + apps := []AppDescriptor{} + for _, window := range backend.windows { + owner := backend.owners[window.handle] + index := slices.IndexFunc(apps, func(app AppDescriptor) bool { return app.Name == owner.Name }) + if index < 0 { + index = len(apps) + apps = append(apps, owner) + } + apps[index].Windows = append(apps[index].Windows, ProviderAppWindow{ + Title: window.title, Role: window.role, Handle: window.handle, + ProviderBusName: owner.ProviderBusName, ProviderObjectPath: window.path, + ProcessGeneration: owner.ProcessGeneration, Active: window.active, InputTarget: true, + }) + } + return ProviderAccessibilityResult{OK: true, Apps: apps}, nil +} + +func (backend *axActivationBackend) Observe(request ProviderObserveRequest) (ProviderAccessibilityResult, error) { + backend.observeRequests = append(backend.observeRequests, request) + if _, exists := backend.owners[request.WindowHandle]; !exists { + return ProviderAccessibilityResult{}, fmt.Errorf("unexpected observation handle %q", request.WindowHandle) + } + return ProviderAccessibilityResult{OK: true, Snapshot: backend.snapshot(request.WindowHandle)}, nil +} + +func (backend *axActivationBackend) ResolveAndAct(request ProviderActionRequest) (ProviderAccessibilityResult, error) { + backend.actionRequests = append(backend.actionRequests, request) + handle := request.WindowHandle + if request.Target != nil && request.Target.WindowHandle != "" { + if handle != "" && handle != request.Target.WindowHandle { + return ProviderAccessibilityResult{}, fmt.Errorf("input and target disagree: %q, %q", handle, request.Target.WindowHandle) + } + handle = request.Target.WindowHandle + } + backend.deliveries = append(backend.deliveries, string(request.Intent)+"@"+handle) + if _, exists := backend.owners[handle]; !exists { + return ProviderAccessibilityResult{}, fmt.Errorf("unexpected input handle %q", handle) + } + if backend.transitionAfterInput != "" { + handle = backend.transitionAfterInput + backend.transitionAfterInput = "" + } + if backend.inputModal != nil { + modal := *backend.inputModal + backend.windows = append(backend.windows, modal) + backend.owners[modal.handle] = backend.owners[handle] + handle = modal.handle + backend.inputModal = nil + } + return ProviderAccessibilityResult{ + OK: true, Snapshot: backend.snapshot(handle), + Mutation: &ProviderMutationFacts{InputEventsEmitted: true, DesktopStateMutated: true}, + ActionEvidence: &ProviderActionEvidence{InputEmitted: true, Route: "semantic_action"}, + }, nil +} + +func (backend *axActivationBackend) ActOnWindow(request ProviderWindowActionRequest) (ProviderWindowResult, error) { + backend.windowRequests = append(backend.windowRequests, request) + handle := request.WindowHandle + if request.Target != nil && request.Target.WindowHandle != "" { + if handle != "" && handle != request.Target.WindowHandle { + return ProviderWindowResult{}, fmt.Errorf("activation and target disagree: %q, %q", handle, request.Target.WindowHandle) + } + handle = request.Target.WindowHandle + } + backend.deliveries = append(backend.deliveries, string(request.Action)+"@"+handle) + if handle == backend.failActivate { + return ProviderWindowResult{ + OK: false, Error: "activation failed before input", + Failure: &ProviderFailure{Code: "input_failed", Phase: "emit", Subject: "window", FactsComplete: true}, + Mutation: &ProviderMutationFacts{}, + }, nil + } + if request.Action != "activate" { + return ProviderWindowResult{}, fmt.Errorf("unexpected window action %q", request.Action) + } + if _, exists := backend.owners[handle]; !exists { + return ProviderWindowResult{}, fmt.Errorf("unexpected activation handle %q", handle) + } + if backend.activationModal != nil { + modal := *backend.activationModal + backend.windows = append(backend.windows, modal) + backend.owners[modal.handle] = backend.owners[handle] + handle = modal.handle + backend.activationModal = nil + } + for index := range backend.windows { + backend.windows[index].active = backend.windows[index].handle == handle + } + return ProviderWindowResult{ + OK: true, Snapshot: backend.snapshot(handle), + Mutation: &ProviderMutationFacts{InputEventsEmitted: true, DesktopStateMutated: true}, + ActionEvidence: &ProviderActionEvidence{InputEmitted: true, Route: "native_window_command"}, + }, nil +} + +type axActivationRefs struct { + window string + button string + entry string +} + +func axActivationService(t *testing.T, crossApp bool) (*Service, *axActivationBackend, axActivationRefs, axActivationRefs) { + t.Helper() + first := AppDescriptor{Name: "Editor", PID: 10, ProviderBusName: ":1.84", + ProcessGeneration: "pid:10:start:100", InputCapable: true} + second := first + if crossApp { + second = AppDescriptor{Name: "Viewer", PID: 20, ProviderBusName: ":1.85", + ProcessGeneration: "pid:20:start:200", InputCapable: true} + } + backend := &axActivationBackend{ + windows: []reportWindow{ + reportMain(), + {handle: reportOtherHandle, role: "frame", title: "Notes", path: "/window/2"}, + }, + owners: map[string]AppDescriptor{reportMainHandle: first, reportOtherHandle: second}, + } + service := NewServiceWithBackend(backend) + listed := service.callTool(contract.PublicToolAX, map[string]any{"mode": "apps"}) + if listed.IsError { + t.Fatalf("list fixture windows: %s", ResultText(listed)) + } + refs := make([]axActivationRefs, 2) + for index, window := range backend.windows { + match := regexp.MustCompile(`(?m)^\s*(w[0-9]+)[^\n]*"` + regexp.QuoteMeta(window.title) + `"`).FindStringSubmatch(ResultText(listed)) + if match == nil { + t.Fatalf("missing window %q in %s", window.title, ResultText(listed)) + } + refs[index].window = match[1] + observed := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "observe", "window_ref": match[1], "observe": "tree", + }) + if observed.IsError { + t.Fatalf("observe fixture window: %s", ResultText(observed)) + } + for name, destination := range map[string]*string{"OK": &refs[index].button, "Size": &refs[index].entry} { + match := regexp.MustCompile(`"` + name + `"[^\n]*\[ref=(r[0-9]+)\]`).FindStringSubmatch(ResultText(observed)) + if match == nil { + t.Fatalf("missing element %q in %s", name, ResultText(observed)) + } + *destination = match[1] + } + } + return service, backend, refs[0], refs[1] +} + +func axActivationEvidence(t *testing.T, result ToolCallResult, windows ...string) { + t.Helper() + batch := result.StructuredContent + if nested, exists := batch["a11y_batch_result"]; exists { + var valid bool + batch, valid = nested.(map[string]any) + if !valid { + t.Fatalf("invalid batch evidence: %#v", nested) + } + } + actions, _ := batch["actions"].([]map[string]any) + if len(actions) != len(windows) { + t.Fatalf("action evidence count = %d, want %d: %#v", len(actions), len(windows), batch) + } + for index, window := range windows { + if actions[index]["window_ref"] != window { + t.Errorf("action %d window evidence = %#v, want %s", index, actions[index], window) + } + } +} + +func TestAXBatchActivationSwitchesTarget(t *testing.T) { + for _, scenario := range []struct { + name string + crossApp, byTitle, explicit bool + }{ + {name: "same_app_ref"}, + {name: "same_app_title_overrides_window", byTitle: true, explicit: true}, + {name: "cross_app_ref_overrides_window", crossApp: true, explicit: true}, + {name: "cross_app_title", crossApp: true, byTitle: true}, + } { + t.Run(scenario.name, func(t *testing.T) { + service, backend, first, second := axActivationService(t, scenario.crossApp) + activation := map[string]any{"type": "activate", "ref": second.window} + if scenario.byTitle { + activation = map[string]any{"type": "activate", "title": "Notes"} + } + args := map[string]any{"mode": "act", "observe": "tree", "actions": []any{ + map[string]any{"type": "click", "ref": first.button}, + activation, + map[string]any{"type": "keypress", "keys": []any{"ENTER"}}, + map[string]any{"type": "type", "text": "untargeted"}, + map[string]any{"type": "click", "ref": second.button}, + map[string]any{"type": "keypress", "ref": second.entry, "keys": []any{"TAB"}}, + map[string]any{"type": "type", "ref": second.entry, "text": "targeted"}, + }} + if scenario.explicit { + args["window_ref"] = first.window + } + result := service.callTool(contract.PublicToolAX, args) + if result.IsError { + t.Fatalf("explicit switch rejected: %s", ResultText(result)) + } + want := []string{"click@" + reportMainHandle, "activate@" + reportOtherHandle, + "keypress@" + reportOtherHandle, "type@" + reportOtherHandle, "click@" + reportOtherHandle, + "keypress@" + reportOtherHandle, "type@" + reportOtherHandle} + if !slices.Equal(backend.deliveries, want) { + t.Errorf("delivery order/handles = %v, want %v", backend.deliveries, want) + } + axActivationEvidence(t, result, first.window, second.window, second.window, + second.window, second.window, second.window, second.window) + owner := backend.owners[reportOtherHandle] + snapshot := service.currentSnapshot(owner.Name) + if snapshot == nil || snapshot.WindowHandle != reportOtherHandle { + t.Fatalf("final snapshot does not belong to B: %#v", snapshot) + } + if !strings.Contains(ResultText(result), `window: "Notes" [ref=`+second.window+`]`) { + t.Errorf("final observation does not identify B: %s", ResultText(result)) + } + if result.StructuredContent["final_observation_returned"] != true { + t.Errorf("missing final observation evidence: %#v", result.StructuredContent) + } + }) + } +} + +func TestAXBatchActivationReturnsToOriginalWindow(t *testing.T) { + for _, crossApp := range []bool{false, true} { + t.Run(fmt.Sprintf("cross_app_%t", crossApp), func(t *testing.T) { + service, backend, first, second := axActivationService(t, crossApp) + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "window_ref": first.window, "observe": "tree", "actions": []any{ + map[string]any{"type": "activate", "ref": first.window}, + map[string]any{"type": "click", "ref": first.button}, + map[string]any{"type": "activate", "ref": second.window}, + map[string]any{"type": "click", "ref": second.button}, + map[string]any{"type": "activate", "ref": first.window}, + map[string]any{"type": "click", "ref": first.button}, + }, + }) + if result.IsError { + t.Fatalf("A -> B -> A rejected: %s", ResultText(result)) + } + want := []string{"activate@" + reportMainHandle, "click@" + reportMainHandle, + "activate@" + reportOtherHandle, "click@" + reportOtherHandle, + "activate@" + reportMainHandle, "click@" + reportMainHandle} + if !slices.Equal(backend.deliveries, want) { + t.Errorf("delivery order/handles = %v, want %v", backend.deliveries, want) + } + axActivationEvidence(t, result, first.window, first.window, second.window, + second.window, first.window, first.window) + if !strings.Contains(ResultText(result), `window: "Report" [ref=`+first.window+`]`) { + t.Errorf("final observation did not return to A: %s", ResultText(result)) + } + }) + } +} + +func TestAXBatchActivationFirstActionOverridesWindow(t *testing.T) { + service, backend, first, second := axActivationService(t, false) + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "window_ref": first.window, "observe": "tree", "actions": []any{ + map[string]any{"type": "activate", "ref": second.window}, + map[string]any{"type": "keypress", "keys": []any{"ENTER"}}, + map[string]any{"type": "type", "text": "only B receives input"}, + }, + }) + if result.IsError { + t.Fatalf("first activation did not override window_ref: %s", ResultText(result)) + } + want := []string{"activate@" + reportOtherHandle, "keypress@" + reportOtherHandle, "type@" + reportOtherHandle} + if !slices.Equal(backend.deliveries, want) { + t.Errorf("first activation delivery targets = %v, want %v", backend.deliveries, want) + } + axActivationEvidence(t, result, second.window, second.window, second.window) + if !strings.Contains(ResultText(result), `window: "Notes" [ref=`+second.window+`]`) { + t.Errorf("final observation does not identify B: %s", ResultText(result)) + } +} + +func TestAXBatchActivationBindsInitialRefLessInputToFocus(t *testing.T) { + for _, crossApp := range []bool{false, true} { + t.Run(fmt.Sprintf("cross_app_%t", crossApp), func(t *testing.T) { + service, backend, first, second := axActivationService(t, crossApp) + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "observe": "tree", "actions": []any{ + map[string]any{"type": "keypress", "keys": []any{"ENTER"}}, + map[string]any{"type": "type", "text": "focused A"}, + map[string]any{"type": "activate", "ref": second.window}, + map[string]any{"type": "keypress", "keys": []any{"TAB"}}, + map[string]any{"type": "type", "text": "activated B"}, + }, + }) + if result.IsError { + t.Fatalf("focus-bound initial segment rejected: %s", ResultText(result)) + } + want := []string{"keypress@" + reportMainHandle, "type@" + reportMainHandle, + "activate@" + reportOtherHandle, "keypress@" + reportOtherHandle, "type@" + reportOtherHandle} + if !slices.Equal(backend.deliveries, want) { + t.Errorf("focus/activation delivery targets = %v, want %v", backend.deliveries, want) + } + axActivationEvidence(t, result, first.window, first.window, second.window, second.window, second.window) + if !strings.Contains(ResultText(result), `window: "Notes" [ref=`+second.window+`]`) { + t.Errorf("final observation does not identify B: %s", ResultText(result)) + } + }) + } +} + +func TestAXBatchActivationStopsWhenActivationOpensModal(t *testing.T) { + service, backend, first, second := axActivationService(t, true) + backend.activationModal = &reportWindow{ + handle: reportDialogHandle, role: "dialog", title: "Confirm changes", path: "/window/3", + } + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "observe": "tree", "actions": []any{ + map[string]any{"type": "click", "ref": first.button}, + map[string]any{"type": "activate", "ref": second.window}, + map[string]any{"type": "keypress", "keys": []any{"ENTER"}}, + map[string]any{"type": "type", "text": "must not reach modal"}, + }, + }) + public := PrepareAXEvidence(result).StructuredContent + if !result.IsError || public["code"] != contract.AXCodeWindowChanged { + t.Errorf("unexpected modal lost window_changed guard: %#v\n%s", public, ResultText(result)) + } + want := []string{"click@" + reportMainHandle, "activate@" + reportOtherHandle} + if !slices.Equal(backend.deliveries, want) { + t.Errorf("input continued after activation opened a modal: %v, want %v", backend.deliveries, want) + } + axActivationEvidence(t, result, first.window, second.window) + if public["completed"] != 2 || result.StructuredContent["stopped_before_action_index"] != 2 { + t.Errorf("modal stop lost completed/next step: %#v", result.StructuredContent) + } +} + +func TestAXBatchActivationRejectsWrongWindowRefsBeforeInput(t *testing.T) { + for _, scenario := range []struct { + name, field string + afterSwitch bool + }{ + {"old_ref_after_switch", "ref", true}, + {"early_ref_before_switch", "ref", false}, + {"old_wait_ref_after_switch", "wait", true}, + {"early_wait_ref_before_switch", "wait", false}, + {"old_to_ref_after_switch", "to_ref", true}, + {"early_to_ref_before_switch", "to_ref", false}, + } { + t.Run(scenario.name, func(t *testing.T) { + service, backend, first, second := axActivationService(t, false) + current, wrong := first, second + actions := []any{map[string]any{"type": "click", "ref": first.button}} + if scenario.afterSwitch { + current, wrong = second, first + actions = append(actions, map[string]any{"type": "activate", "ref": second.window}) + } + badIndex := len(actions) + field := "ref" + badAction := map[string]any{"type": "click", "ref": wrong.button} + switch scenario.field { + case "wait": + badAction = map[string]any{"type": "wait", "ref": wrong.entry, "state": "visible", "timeout_ms": 1} + case "to_ref": + field = "to_ref" + badAction = map[string]any{"type": "select", "ref": current.entry, "to_ref": wrong.entry} + } + actions = append(actions, badAction) + if !scenario.afterSwitch { + actions = append(actions, map[string]any{"type": "activate", "ref": second.window}) + } + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "actions": actions, + }) + path := fmt.Sprintf("actions[%d].%s", badIndex, field) + public := PrepareAXEvidence(result).StructuredContent + if !result.IsError || public["path"] != path { + t.Errorf("wrong-window ref must fail at %s: %#v", path, public) + } + if len(backend.actionRequests) != 0 || len(backend.windowRequests) != 0 || len(backend.desktopRequests) != 0 { + t.Errorf("ownership preflight reached input backend: actions=%#v, windows=%#v, desktop=%#v", + backend.actionRequests, backend.windowRequests, backend.desktopRequests) + } + if public["input_emitted"] != false || public["completed"] != 0 || public["code"] != contract.AXCodeInvalidArgument { + t.Errorf("wrong-window rejection evidence: %#v", public) + } + }) + } +} + +func TestAXBatchActivationFailureStopsFollowingInput(t *testing.T) { + service, backend, first, second := axActivationService(t, true) + backend.failActivate = reportOtherHandle + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "observe": "tree", "actions": []any{ + map[string]any{"type": "click", "ref": first.button}, + map[string]any{"type": "activate", "ref": second.window}, + map[string]any{"type": "keypress", "keys": []any{"ENTER"}}, + map[string]any{"type": "type", "ref": second.entry, "text": "must not run"}, + }, + }) + if !result.IsError || !strings.Contains(ResultText(result), "activation failed before input") { + t.Errorf("activation failure missing: %s", ResultText(result)) + } + want := []string{"click@" + reportMainHandle, "activate@" + reportOtherHandle} + if !slices.Equal(backend.deliveries, want) { + t.Errorf("failure did not stop immediately: %v, want %v", backend.deliveries, want) + } + axActivationEvidence(t, result, first.window, second.window) + batch, _ := result.StructuredContent["a11y_batch_result"].(map[string]any) + if batch["failed_action_index"] != 1 || batch["actions_completed_without_call_error"] != 1 { + t.Errorf("failure location/completed count: %#v", batch) + } +} + +func TestAXBatchActivationKeepsImplicitWindowChangeGuard(t *testing.T) { + service, backend, first, _ := axActivationService(t, false) + backend.transitionAfterInput = reportOtherHandle + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "window_ref": first.window, "observe": "tree", "actions": []any{ + map[string]any{"type": "click", "ref": first.button}, + map[string]any{"type": "keypress", "keys": []any{"ENTER"}}, + }, + }) + public := PrepareAXEvidence(result).StructuredContent + if !result.IsError || public["code"] != contract.AXCodeWindowChanged { + t.Errorf("implicit switch lost window_changed guard: %#v\n%s", public, ResultText(result)) + } + if want := []string{"click@" + reportMainHandle}; !slices.Equal(backend.deliveries, want) { + t.Errorf("input continued after implicit switch: %v", backend.deliveries) + } +} + +func TestAXBatchActivationRejectsOldWindowCloseAfterWait(t *testing.T) { + service, backend, first, _ := axActivationService(t, false) + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "actions": []any{ + map[string]any{"type": "activate", "ref": first.window}, + map[string]any{"type": "wait", "window": "Notes"}, + map[string]any{"type": "close", "ref": first.window}, + }, + }) + public := PrepareAXEvidence(result).StructuredContent + if !result.IsError || public["code"] != contract.AXCodeInvalidArgument || + public["path"] != "actions[2].ref" || public["completed"] != 2 { + t.Fatalf("old window close was not rejected after wait: %#v", public) + } + if len(backend.deliveries) != 1 { + t.Fatalf("old window received input after wait: %v", backend.deliveries) + } +} + +func TestAXBatchActivationFinalObservationIdentifiesModal(t *testing.T) { + service, backend, first, second := axActivationService(t, true) + backend.inputModal = &reportWindow{ + handle: reportDialogHandle, role: "dialog", title: "Opened document", path: "/window/3", active: true, + } + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "observe": "tree", "actions": []any{ + map[string]any{"type": "activate", "ref": first.window}, + map[string]any{"type": "activate", "ref": second.window}, + map[string]any{"type": "keypress", "keys": []any{"CTRL", "O"}}, + }, + }) + if result.IsError { + t.Fatal(ResultText(result)) + } + snapshot := service.currentSnapshot(backend.owners[reportDialogHandle].Name) + if result.StructuredContent["final_observation_window_ref"] != snapshot.publicWindowRef || + !strings.Contains(ResultText(result), "final "+snapshot.publicWindowRef) { + t.Fatalf("modal observation has the wrong identity: %#v\n%s", result.StructuredContent, ResultText(result)) + } + axActivationEvidence(t, result, first.window, second.window, second.window) +} + +func TestAXBatchActivationFinalWaitDoesNotInventWindowLifecycle(t *testing.T) { + service, _, first, second := axActivationService(t, true) + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "observe": "tree", "actions": []any{ + map[string]any{"type": "activate", "ref": first.window}, + map[string]any{"type": "wait", "window": "Notes"}, + }, + }) + text := ResultText(result) + if result.IsError || strings.Contains(text, "opened "+second.window) || strings.Contains(text, "closed "+first.window) { + t.Fatalf("wait fabricated window lifecycle: %s", text) + } + if !strings.Contains(text, "final "+second.window) { + t.Fatalf("wait report lost final window: %s", text) + } + axActivationEvidence(t, result, first.window, second.window) +} diff --git a/computer-use/internal/core/ax_batch_context.go b/computer-use/internal/core/ax_batch_context.go index df5f5e7..e40c1bb 100644 --- a/computer-use/internal/core/ax_batch_context.go +++ b/computer-use/internal/core/ax_batch_context.go @@ -34,16 +34,65 @@ const ( axWindowUnknown axWindowContinuity = "unknown" ) -// deriveAXBatchContext binds a batch to one logical window. The window -// comes from the actions' rN/wN refs, from window_ref, or, for a batch whose -// actions carry no ref, from the window that holds keyboard focus. A -// window_ref that names the same window as the refs is accepted; a -// different one is rejected. focus is the focused window read before any -// input; it is used only when nothing else names a window. +func axBatchWindowEnd(actions []axBatchAction, start int) int { + for index := start + 1; index < len(actions); index++ { + if actions[index].actionType == "activate" { + return index + } + } + return len(actions) +} + +func (s *Service) deriveAXBatchContexts( + explicitWindowRef string, + actions []axBatchAction, + focus *axDesktopWindow, +) ([]*AXBatchContext, error) { + contexts := make([]*AXBatchContext, len(actions)) + for start := 0; start < len(actions); { + end := axBatchWindowEnd(actions, start) + windowRef := explicitWindowRef + if actions[start].actionType == "activate" { + windowRef = actions[start].targetRef + } + context, err := s.deriveAXWindowContext(windowRef, actions[start:end], focus, start) + if err != nil { + return nil, err + } + if err := s.validateAXNoTreeSegment(context, actions[start:end], start); err != nil { + return nil, err + } + for index := start; index < end; index++ { + contexts[index] = context + if actions[index].wait != nil && actions[index].wait.ref != "" { + if err := s.preflightAXWaitRef(context, actions[index].wait.ref); err != nil { + return nil, axBatchRefError(index, "ref", err) + } + } + if actions[index].toRef != "" { + if err := s.preflightAXWaitRef(context, actions[index].toRef); err != nil { + return nil, axBatchRefError(index, "to_ref", err) + } + } + } + start = end + } + return contexts, nil +} + func (s *Service) deriveAXBatchContext( explicitWindowRef string, actions []axBatchAction, focus *axDesktopWindow, +) (*AXBatchContext, error) { + return s.deriveAXWindowContext(explicitWindowRef, actions, focus, 0) +} + +func (s *Service) deriveAXWindowContext( + explicitWindowRef string, + actions []axBatchAction, + focus *axDesktopWindow, + actionOffset int, ) (*AXBatchContext, error) { context := &AXBatchContext{} bindWindow := func(ref, handle, appRef, appIdentity string) error { @@ -51,9 +100,9 @@ func (s *Service) deriveAXBatchContext( handle = strings.TrimSpace(handle) if context.WindowHandle != "" && context.WindowHandle != handle { return axInvalid("", fmt.Sprintf( - "batch spans two windows (%s and %s); one act call works in one window, so split it", - context.WindowRef, ref), - "Split the actions into separate act calls, one window per call.", + "ref belongs to window %s, not the current batch window %s; activate that window before using its refs", + ref, context.WindowRef), + "Use refs from the current window, or activate the other window before acting in it.", ) } if context.WindowRef == "" { @@ -93,7 +142,7 @@ func (s *Service) deriveAXBatchContext( if err == nil { err = bindWindow(entry.LogicalWindowRef, entry.WindowHandle, entry.AppRef, entry.AppIdentity) - if err != nil && explicitWindowRef != "" && index == 0 { + if err != nil && explicitWindowRef != "" && index == 0 && actionOffset == 0 { err = axInvalid("window_ref", fmt.Sprintf("%s belongs to window %s, not window_ref %s; omit window_ref", address, entry.LogicalWindowRef, explicitWindowRef), "Omit window_ref so the action refs determine the window, or use refs from window_ref.") @@ -107,11 +156,7 @@ func (s *Service) deriveAXBatchContext( } } if err != nil { - var argument *axArgError - if errors.As(err, &argument) && argument.path == "" { - argument.path = fmt.Sprintf("actions[%d].ref", index) - } - return nil, fmt.Errorf("actions[%d].ref: %w", index, err) + return nil, axBatchRefError(actionOffset+index, "ref", err) } } if context.WindowHandle == "" && !hasTarget && focus != nil { @@ -143,12 +188,24 @@ func (s *Service) deriveAXBatchContext( return context, nil } +func axBatchRefError(index int, field string, err error) error { + path := fmt.Sprintf("actions[%d].%s", index, field) + var argument *axArgError + if errors.As(err, &argument) && argument.path == "" { + argument.path = path + } + return fmt.Errorf("%s: %w", path, err) +} + func (s *Service) axBatchWindowContinuity( context *AXBatchContext, ) axWindowContinuity { if s == nil || context == nil || strings.TrimSpace(context.WindowHandle) == "" { return axWindowUnknown } + if continuity, native := s.axNoTreeBatchWindowContinuity(context); native { + return continuity + } snapshot := s.currentSnapshot(context.AppToken) if snapshot == nil { return axWindowUnknown @@ -182,6 +239,27 @@ func (s *Service) axBatchWindowContinuity( return axWindowSame } +func (s *Service) validateAXActionWindow(context *AXBatchContext, action axBatchAction, index int) error { + for _, target := range []struct{ field, ref string }{{"ref", action.targetRef}, {"to_ref", action.toRef}} { + if isShortAXRef(target.ref) { + if err := s.preflightAXWaitRef(context, target.ref); err != nil { + return axBatchRefError(index, target.field, err) + } + } else if isShortAXWindowRef(target.ref) && action.actionType != "activate" { + entry, err := s.resolveShortAXWindowRefOwned(target.ref) + if err != nil { + return axBatchRefError(index, target.field, err) + } + if entry.WindowHandle != context.WindowHandle { + return axBatchRefError(index, target.field, axInvalid("", fmt.Sprintf( + "%s belongs to another window, not the current batch window %s", target.ref, context.WindowRef), + "Activate that window before acting in it.")) + } + } + } + return nil +} + // axBatchFocusChanged reports whether keyboard focus left the window a // focus-bound batch was bound to, and if so returns the stop result. func (s *Service) axBatchFocusChanged( diff --git a/computer-use/internal/core/ax_batch_observation.go b/computer-use/internal/core/ax_batch_observation.go index 5e0cd6d..aefba6c 100644 --- a/computer-use/internal/core/ax_batch_observation.go +++ b/computer-use/internal/core/ax_batch_observation.go @@ -91,6 +91,9 @@ func (s *Service) axBatchFinalObservation( if context == nil || obs.resolvedMode() == contract.ObserveNone { return ToolCallResult{} } + if window := s.noTreeWindowEntry(context.WindowRef); window != nil { + return axNoTreeBatchObservation(window) + } request := AXServiceRequest{ Operation: axServiceObserve, App: context.AppToken, diff --git a/computer-use/internal/core/ax_batch_result.go b/computer-use/internal/core/ax_batch_result.go index 05384a2..664f51b 100644 --- a/computer-use/internal/core/ax_batch_result.go +++ b/computer-use/internal/core/ax_batch_result.go @@ -7,20 +7,49 @@ import ( "strings" ) +func axBatchFailureHint(context *AXBatchContext, evidence map[string]any) string { + if hint, _ := evidence[contract.AXPublicErrorHintKey].(string); hint != "" { + return "" + } + if evidence["focus_preparation"] != nil || evidence["pointer_preparation"] != nil { + return "" + } + switch evidence["failure_code"] { + case "unsupported_action": + return GUIScreenshotCall + case "invalid_argument", "input_failed": + if context != nil && isShortAXWindowRef(context.WindowRef) { + return contract.CompactJSON(map[string]any{"mode": contract.AXModeObserve, "window_ref": context.WindowRef}) + } + return GUIScreenshotCall + } + return "" +} + func prependAXBatchSummary( result ToolCallResult, summary string, facts map[string]any, ) ToolCallResult { + if result.StructuredContent == nil { + result.StructuredContent = map[string]any{} + } + if result.IsError { + result.StructuredContent[contract.AXPublicErrorDetailKey] = + axPublicErrorMessage(result.StructuredContent, result.Content) + if axPublicErrorCode(result.StructuredContent) == contract.AXCodeInvalidArgument && + result.StructuredContent[contract.AXPublicErrorPathKey] == nil { + if index, ok := IntEvidence(facts["failed_action_index"]); ok { + result.StructuredContent[contract.AXPublicErrorPathKey] = fmt.Sprintf("actions[%d]", index) + } + } + } if strings.TrimSpace(summary) != "" { result.Content = append( []ContentItem{{Type: "text", Text: summary}}, result.Content..., ) } - if result.StructuredContent == nil { - result.StructuredContent = map[string]any{} - } result.StructuredContent["a11y_batch_result"] = facts if actions, ok := facts["actions"].([]map[string]any); ok { emitted, partial, unknown := batchMutationSummary(actions) @@ -54,7 +83,8 @@ func compactAXActionContent(result ToolCallResult) ToolCallResult { kept := make([]string, 0, len(lines)) for _, line := range lines { trimmed := strings.TrimSpace(line) - if strings.HasPrefix(trimmed, "Note:") { + if strings.HasPrefix(trimmed, "Note:") && + !strings.HasPrefix(strings.TrimSpace(strings.TrimPrefix(trimmed, "Note:")), "[a11y][synthesis]") { drop := false for _, marker := range routine { if strings.Contains(trimmed, marker) { diff --git a/computer-use/internal/core/ax_batch_result_test.go b/computer-use/internal/core/ax_batch_result_test.go new file mode 100644 index 0000000..1092195 --- /dev/null +++ b/computer-use/internal/core/ax_batch_result_test.go @@ -0,0 +1,188 @@ +package core + +import ( + "strings" + "testing" + + "github.com/microsoft/Hybrid-Computer-Use/computer-use/internal/contract" +) + +type envelopeFailureBackend struct { + *reportBackend + failure ProviderAccessibilityResult + failAt int + attempts int +} + +func (backend *envelopeFailureBackend) ResolveAndAct(request ProviderActionRequest) (ProviderAccessibilityResult, error) { + backend.attempts++ + if backend.attempts == backend.failAt { + return backend.failure, nil + } + return backend.reportBackend.ResolveAndAct(request) +} + +func TestGlobalInputForbiddenKeepsCauseAndBatchFacts(t *testing.T) { + for _, unknown := range []bool{false, true} { + for _, failAt := range []int{1, 2} { + service, desktop := reportService(t, reportMain()) + _, refs := reportObserve(t, service, "OK") + code, phase := contract.AXCodeGlobalInputForbidden, "precondition" + if unknown { + code, phase = contract.AXCodeEffectUnknown, "emission" + } + backend := &envelopeFailureBackend{ + reportBackend: desktop, failAt: failAt, + failure: ProviderAccessibilityResult{ + Error: "semantic route unavailable or unverified; no global fallback", + Failure: &ProviderFailure{Code: code, Phase: phase, FactsComplete: true}, + Mutation: &ProviderMutationFacts{InputEventsEmitted: unknown, MutationUnknown: unknown}, + }, + } + service.Backend = &globalInputPolicyBackend{backend, "linux"} + result := PrepareAXEvidence(service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "actions": []any{ + map[string]any{"type": "click", "ref": refs["OK"], "allow_global_input": false}, + map[string]any{"type": "click", "ref": refs["OK"], "allow_global_input": false}, + }, + })) + public := result.StructuredContent + if !result.IsError || public["code"] != code || public["completed"] != failAt-1 || + public["input_emitted"] != (unknown || failAt > 1) { + t.Fatalf("failed action/prefix evidence lost: %#v, %s", public, ResultText(result)) + } + if backend.attempts != failAt { + t.Fatalf("failure retried or later action ran: %d attempts", backend.attempts) + } + } + } +} + +type synthesisNoteBackend struct { + *reportBackend + note string +} + +func (backend *synthesisNoteBackend) ResolveAndAct(request ProviderActionRequest) (ProviderAccessibilityResult, error) { + result, err := backend.reportBackend.ResolveAndAct(request) + if len(backend.actionRequests) == 1 { + result.Notes = []string{backend.note} + } + return result, err +} + +func TestAXBatchRetainsEarlierSynthesisWarning(t *testing.T) { + for _, observe := range []string{"none", "diff", "tree"} { + service, desktop := reportService(t, reportMain()) + _, refs := reportObserve(t, service, "OK", "Size") + note := "[a11y][synthesis] Synthesized click. Global input may change focus; verify the effect." + service.Backend = &synthesisNoteBackend{desktop, note} + result := PrepareAXEvidence(service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "observe": observe, "actions": []any{ + map[string]any{"type": "click", "ref": refs["OK"]}, + map[string]any{"type": "set_value", "ref": refs["Size"], "value": "14"}, + }, + })) + if result.IsError || strings.Count(ResultText(result), note) != 1 { + t.Fatalf("%s lost or duplicated earlier synthesis warning: %s", observe, ResultText(result)) + } + } +} + +func TestRuntimeFailureEnvelopeKeepsCauseAndBatchFacts(t *testing.T) { + for _, failure := range []struct { + code, phase, reason, publicCode string + }{ + {"invalid_argument", "precondition", `select_text_not_found: "missing" is not in the element's text; no input was emitted.`, contract.AXCodeInvalidArgument}, + {"invalid_argument", "precondition", "select_unavailable: to_ref must be a later text sibling of ref in the same container; no input was emitted.", contract.AXCodeInvalidArgument}, + {"unsupported_action", "precondition", "This control has no verified text editor; no text input was emitted.", contract.AXCodeInvalidArgument}, + {"input_failed", "emit", "input_failed: the accessibility runtime failed before input.", contract.AXCodeRuntimeError}, + } { + for _, failAt := range []int{1, 2} { + t.Run(failure.reason+itoa(failAt), func(t *testing.T) { + service, desktop := reportService(t, reportMain()) + window, refs := reportObserve(t, service, "Size") + backend := &envelopeFailureBackend{reportBackend: desktop, failAt: failAt, + failure: ProviderAccessibilityResult{OK: false, Error: failure.reason, + Failure: &ProviderFailure{Code: failure.code, Phase: failure.phase, Subject: "element", FactsComplete: true}, + Mutation: &ProviderMutationFacts{}}, + } + service.Backend = backend + result := PrepareAXEvidence(service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "window_ref": window, "observe": "none", + "actions": []any{ + map[string]any{"type": "set_value", "ref": refs["Size"], "value": "16"}, + map[string]any{"type": "set_value", "ref": refs["Size"], "value": "18"}, + map[string]any{"type": "set_value", "ref": refs["Size"], "value": "20"}, + }, + })) + public := result.StructuredContent + wantCode := failure.publicCode + if failAt > 1 { + wantCode = contract.AXCodeEffectUnknown + } + if !result.IsError || public["code"] != wantCode || public["message"] != failure.reason { + t.Errorf("lost failure cause: %#v\n%s", public, ResultText(result)) + } + if wantCode == contract.AXCodeInvalidArgument && public["path"] != "actions[0]" { + t.Errorf("argument failure must name the failed action: %#v", public) + } + if backend.attempts != failAt || public["completed"] != failAt-1 || public["requested"] != 3 || + public["input_emitted"] != (failAt > 1) || public["failure_code"] != failure.code { + t.Fatalf("lost batch stop/input facts: attempts=%d %#v", backend.attempts, public) + } + batch, _ := public["batch"].(map[string]any) + if batch["failed_action_index"] != failAt-1 || !strings.Contains(ResultText(result), "Stopped at action") { + t.Fatalf("lost batch context: %#v %s", batch, ResultText(result)) + } + hint, _ := public["hint"].(string) + if failure.code == "unsupported_action" { + if hint != GUIScreenshotCall { + t.Errorf("unsupported semantic action needs visual recovery: %q", hint) + } + } else { + args := decodeJSONForTest(t, hint) + if args["mode"] != "observe" || args["window_ref"] != window { + t.Errorf("recovery must inspect the failed target's window: %s", hint) + } + if _, err := DecodeAXRequest(args); err != nil { + t.Errorf("invalid recovery call: %v", err) + } + } + }) + } + } +} + +func TestBatchFailureHintPreservesExplicitRecoveryAndPreparationStops(t *testing.T) { + context := &AXBatchContext{WindowRef: "w7"} + for _, evidence := range []map[string]any{ + {"failure_code": "invalid_argument", contract.AXPublicErrorHintKey: "specific recovery"}, + {"failure_code": "input_failed", "focus_preparation": "unknown"}, + {"failure_code": "input_failed", "pointer_preparation": "unknown"}, + {"failure_code": "stale_element_ref"}, + } { + if hint := axBatchFailureHint(context, evidence); hint != "" { + t.Fatalf("overrode existing recovery policy: %s %#v", hint, evidence) + } + } + if hint := axBatchFailureHint(nil, map[string]any{"failure_code": "input_failed"}); hint != GUIScreenshotCall { + t.Fatalf("unaddressable window must not invent a ref: %s", hint) + } +} + +func TestProviderArgumentFailureWithUncertainInputIsEffectUnknown(t *testing.T) { + for _, code := range []string{"invalid_argument", "unsupported_action", "input_failed"} { + for _, evidence := range []map[string]any{ + {"partial_input_possible": true}, + {"mutation_unknown": true}, + {"input_events_emitted": true}, + } { + evidence["failure_code"] = code + evidence["failure_phase"] = "precondition" + if got := axPublicErrorCode(evidence); got != contract.AXCodeEffectUnknown { + t.Errorf("uncertain input must not become a no-input error: %s %#v", got, evidence) + } + } + } +} diff --git a/computer-use/internal/core/ax_batch_test.go b/computer-use/internal/core/ax_batch_test.go index e685950..93ab3e1 100644 --- a/computer-use/internal/core/ax_batch_test.go +++ b/computer-use/internal/core/ax_batch_test.go @@ -8,6 +8,147 @@ import ( "github.com/microsoft/Hybrid-Computer-Use/computer-use/internal/contract" ) +type globalInputPolicyBackend struct { + PlatformBackend + platform string +} + +func (backend *globalInputPolicyBackend) HostInfo() HostBackendInfo { + return HostBackendInfo{Platform: backend.platform} +} + +func TestAXGlobalInputPolicyRejectsWholeInvalidBatch(t *testing.T) { + tests := []struct { + name, platform string + action map[string]any + }{} + for _, actionType := range contract.AXActionTypes { + if actionType == "click" || actionType == "scroll" { + continue + } + for _, policy := range []bool{false, true} { + tests = append(tests, struct { + name, platform string + action map[string]any + }{actionType + "/" + contract.CompactJSON(policy), "linux", map[string]any{"type": actionType, "allow_global_input": policy}}) + } + } + for _, actionType := range []string{"click", "scroll"} { + for _, value := range []any{nil, "false", 0.0, 1.0, []any{}, map[string]any{}} { + tests = append(tests, struct { + name, platform string + action map[string]any + }{actionType + "/" + contract.CompactJSON(value), "linux", map[string]any{ + "type": actionType, "allow_global_input": value, + }}) + } + for _, platform := range []string{"darwin", "windows", "unknown"} { + for _, policy := range []bool{false, true} { + action := map[string]any{"type": actionType, "ref": "r1", "allow_global_input": policy} + if actionType == "scroll" { + action["direction"] = "down" + } + tests = append(tests, struct { + name, platform string + action map[string]any + }{platform + "/" + actionType + "/" + contract.CompactJSON(policy), platform, action}) + } + } + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + backend := &recordingPlatformBackend{} + service := NewServiceWithBackend(&globalInputPolicyBackend{backend, test.platform}) + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "actions": []any{ + map[string]any{"type": "activate", "title": "Editor"}, test.action, + }, + }) + evidence := axPublicEvidence(result.StructuredContent, result.IsError, result.Content) + if !result.IsError || evidence["code"] != contract.AXCodeInvalidArgument || + evidence["path"] != "actions[1].allow_global_input" || evidence["completed"] != 0 || evidence["input_emitted"] != false { + t.Fatalf("invalid batch was not rejected before input: %#v, %s", evidence, ResultText(result)) + } + if len(backend.calls) != 0 || len(backend.actionRequests) != 0 { + t.Fatalf("invalid batch reached the provider: %v, %v", backend.calls, backend.actionRequests) + } + }) + } +} + +func TestGlobalInputPolicyRejectsWrongToolOrTopLevel(t *testing.T) { + for _, tool := range []string{contract.PublicToolAX, contract.PublicToolGUI} { + for _, topLevel := range []bool{false, true} { + if tool == contract.PublicToolAX && !topLevel { + continue + } + backend := &recordingPlatformBackend{} + service := NewServiceWithBackend(&globalInputPolicyBackend{backend, "linux"}) + action := map[string]any{"type": "click", "x": 1.0, "y": 1.0} + args := map[string]any{"actions": []any{action}} + if tool == contract.PublicToolAX { + args["mode"] = "act" + args["actions"] = []any{map[string]any{"type": "click", "ref": "r1"}} + } + if topLevel { + args["allow_global_input"] = false + } else { + action["allow_global_input"] = false + } + result := service.callTool(tool, args) + if !result.IsError || !strings.Contains(ResultText(result), "allow_global_input") || len(backend.calls) != 0 { + t.Fatalf("unsupported %s policy accepted: %s, calls %v", tool, ResultText(result), backend.calls) + } + } + } +} + +func TestAXGlobalInputPolicyReachesProvider(t *testing.T) { + for _, kind := range []string{"click", "targeted scroll", "window scroll"} { + for _, policy := range []any{nil, false, true} { + t.Run(kind+"/"+contract.CompactJSON(policy), func(t *testing.T) { + service, backend := reportService(t, reportMain()) + service.Backend = &globalInputPolicyBackend{backend, "linux"} + window, refs := reportObserve(t, service, "OK") + action := map[string]any{"type": "scroll", "direction": "down"} + args := map[string]any{"mode": "act", "actions": []any{action}} + if kind == "click" { + action = map[string]any{"type": "click", "ref": refs["OK"]} + args["actions"] = []any{action} + } else if kind == "targeted scroll" { + action["ref"] = refs["OK"] + } else { + args["window_ref"] = window + } + if policy != nil { + action["allow_global_input"] = policy + } + result := service.callTool(contract.PublicToolAX, args) + if result.IsError || len(backend.actionRequests) != 1 { + t.Fatalf("action failed: %s, requests %v", ResultText(result), backend.actionRequests) + } + request := backend.actionRequests[0] + var got *bool + if kind == "click" { + got = request.Click.AllowGlobalInput + } else { + got = request.Scroll.AllowGlobalInput + if request.Scroll.WindowCentre != (kind == "window scroll") { + t.Fatalf("lost scroll target: %#v", request.Scroll) + } + } + if policy == nil { + if got != nil { + t.Fatal("omitted policy was not preserved") + } + } else if got == nil || *got != policy.(bool) { + t.Fatalf("provider lost policy %v: %v", policy, got) + } + }) + } + } +} + func TestCompactAXActionContentDropsRoutineRouteNotes(t *testing.T) { result := compactAXActionContent(ToolCallResult{ Content: []ContentItem{{Type: "text", Text: strings.Join([]string{ @@ -275,7 +416,7 @@ func TestCrossWindowBatchFailsBeforeInput(t *testing.T) { {actionType: "click", targetRef: secondRef}, }, nil, ) - if err == nil || !strings.Contains(err.Error(), "batch spans two windows") { + if err == nil || !strings.Contains(err.Error(), "not the current batch window") { t.Fatalf("expected cross-window rejection, got %v", err) } } diff --git a/computer-use/internal/core/ax_batch_window_error_test.go b/computer-use/internal/core/ax_batch_window_error_test.go index 0fcc2de..00b5038 100644 --- a/computer-use/internal/core/ax_batch_window_error_test.go +++ b/computer-use/internal/core/ax_batch_window_error_test.go @@ -24,9 +24,9 @@ func TestAXCrossWindowPreflightReportsArgumentError(t *testing.T) { map[string]any{"type": "click", "ref": second}, } args := map[string]any{"mode": "act", "actions": actions} - wantPath, wantHint := "actions[1].ref", "separate" + wantPath, wantHint := "actions[1].ref", "activate" if kind == "window ref action" { - actions[1] = map[string]any{"type": "activate", "ref": secondWindow} + actions[1] = map[string]any{"type": "close", "ref": secondWindow} } if kind == "explicit window" { args["window_ref"] = secondWindow diff --git a/computer-use/internal/core/ax_evidence.go b/computer-use/internal/core/ax_evidence.go index a8c9f99..937deea 100644 --- a/computer-use/internal/core/ax_evidence.go +++ b/computer-use/internal/core/ax_evidence.go @@ -84,11 +84,20 @@ func compactAXBatchResult(value any) map[string]any { } } } + windows := map[string]bool{} + for _, action := range rawActions { + if ref, ok := action["window_ref"].(string); ok && ref != "" { + windows[ref] = true + } + } actions := make([]map[string]any, 0, len(rawActions)) for _, action := range rawActions { entry := map[string]any{ "index": action["action_index"], } + if len(windows) > 1 { + entry["window_ref"] = action["window_ref"] + } actionability, _ := action["actionability"].(map[string]any) if actionability != nil { if resolution := strings.TrimSpace(fmt.Sprint( @@ -161,6 +170,10 @@ func compactAXModelEvidence( ) map[string]any { compact := map[string]any{} mode := strings.TrimSpace(fmt.Sprint(full["ax_mode"])) + if page := full["text_page"]; page != nil { + compact["text_page"] = page + compact["text_read"] = full["text_read"] + } if mode == contract.AXModeFind { for _, key := range []string{ "find_match_count", "find_page_start", "find_page_count", "next_cursor", @@ -544,12 +557,13 @@ var axPublicFields = []string{ // axPublicHints are the default corrected calls per error code. var axPublicHints = map[string]string{ - contract.AXCodeStaleRef: `{"mode":"find","window_ref":"wN","query":{"name_contains":"..."}}`, - contract.AXCodeWindowChanged: `{"mode":"observe","window_ref":"wN"}`, - contract.AXCodeWaitTimeout: `{"mode":"observe","window_ref":"wN"}`, - contract.AXCodeEffectUnknown: `{"mode":"observe","window_ref":"wN"}`, - contract.AXCodeAppUnavailable: `{"mode":"apps"}`, - contract.AXCodeRuntimeError: `{"mode":"apps"}`, + contract.AXCodeStaleRef: `{"mode":"find","window_ref":"wN","query":{"name_contains":"..."}}`, + contract.AXCodeWindowChanged: `{"mode":"observe","window_ref":"wN"}`, + contract.AXCodeWaitTimeout: `{"mode":"observe","window_ref":"wN"}`, + contract.AXCodeEffectUnknown: `{"mode":"observe","window_ref":"wN"}`, + contract.AXCodeGlobalInputForbidden: `{"mode":"observe","window_ref":"wN"}`, + contract.AXCodeAppUnavailable: `{"mode":"apps"}`, + contract.AXCodeRuntimeError: `{"mode":"apps"}`, } // axPublicErrorCode maps internal failure facts to one public code. @@ -569,10 +583,24 @@ func axPublicErrorCode(full map[string]any) string { return contract.AXCodeWaitTimeout case failure == "effect_unknown": return contract.AXCodeEffectUnknown + case failure == contract.AXCodeGlobalInputForbidden: + if axEvidenceReportsInputEmission(full) || full["partial_input_possible"] == true || full["mutation_unknown"] == true { + return contract.AXCodeEffectUnknown + } + if phase == "precondition" { + return contract.AXCodeGlobalInputForbidden + } case failure == "ax_app_unavailable" || status == "app_unavailable": return contract.AXCodeAppUnavailable case failure == "ax_app_ambiguous": return contract.AXCodeInvalidArgument + case failure == "invalid_argument", failure == "unsupported_action", failure == "input_failed": + if axEvidenceReportsInputEmission(full) || full["partial_input_possible"] == true || full["mutation_unknown"] == true { + return contract.AXCodeEffectUnknown + } + if phase == "precondition" && failure != "input_failed" { + return contract.AXCodeInvalidArgument + } case strings.HasPrefix(failure, "stale_"), failure == "unknown_ref", failure == "semantic_target_not_found", failure == "semantic_target_ambiguous", @@ -589,6 +617,21 @@ func axPublicErrorCode(full map[string]any) string { return contract.AXCodeRuntimeError } +func axPublicErrorMessage(full map[string]any, content []ContentItem) string { + if message, _ := full[contract.AXPublicErrorDetailKey].(string); message != "" { + return firstLine(strings.TrimSpace(message)) + } + for _, item := range content { + text := strings.TrimSpace(item.Text) + if item.Type != "text" || text == "" || isActReportText(text) || + strings.HasPrefix(text, "title ") || strings.HasPrefix(text, "bound_window: ") { + continue + } + return firstLine(text) + } + return "" +} + // axPublicEvidence returns the leading fields of the AX evidence line: // ok, the public error code with path, message and hint, and input and // progress counts. @@ -600,20 +643,7 @@ func axPublicEvidence(full map[string]any, isError bool, content []ContentItem) if path, _ := full[contract.AXPublicErrorPathKey].(string); path != "" { public["path"] = path } - message, _ := full[contract.AXPublicErrorDetailKey].(string) - if message == "" { - for _, item := range content { - text := strings.TrimSpace(item.Text) - // Window-resolution notes say where input went; they are - // not the error. - if item.Type != "text" || text == "" || isActReportText(text) || - strings.HasPrefix(text, "title ") || strings.HasPrefix(text, "bound_window: ") { - continue - } - message = text - break - } - } + message := axPublicErrorMessage(full, content) coordinateUnverified := full["failure_code"] == "coordinate_target_unverified" if coordinateUnverified { message = "Screen-point ownership could not be verified. Use GUI screenshot to inspect covering windows and the target; refreshing refs alone does not fix ownership. Check completed and current state before sending remaining actions. " + message diff --git a/computer-use/internal/core/ax_find_delivery.go b/computer-use/internal/core/ax_find_delivery.go index fd5008f..db3001b 100644 --- a/computer-use/internal/core/ax_find_delivery.go +++ b/computer-use/internal/core/ax_find_delivery.go @@ -63,6 +63,9 @@ func FindMatchUnit(snapshot *AXSnapshot, record ElementRecord, ordinal int) stri display := findDisplayNode(node) fmt.Fprintf(&out, " - match=%d %s\n", ordinal, renderAXNodeLabel(display)) for _, property := range renderAXProperties(display, AXRenderOptions{}) { + if property[0] == "/text" { + continue + } out.WriteString(renderAXTreePropertyLine(property, 2) + "\n") } } else { diff --git a/computer-use/internal/core/ax_native_window.go b/computer-use/internal/core/ax_native_window.go index 90e5436..fe62feb 100644 --- a/computer-use/internal/core/ax_native_window.go +++ b/computer-use/internal/core/ax_native_window.go @@ -49,93 +49,75 @@ func (s *Service) noTreeWindowEntry(ref string) *AXWindowRefEntry { return entry } -// axNoTreeBatchWindow returns the window without a tree that a batch names -// through window_ref or an activate or close target, or nil. -func (s *Service) axNoTreeBatchWindow(actions []axBatchAction, windowRef string) *AXWindowRefEntry { - if entry := s.noTreeWindowEntry(windowRef); entry != nil { - return entry - } - for _, action := range actions { - if entry := s.noTreeWindowEntry(action.targetRef); entry != nil { - return entry - } +func (s *Service) validateAXNoTreeSegment(context *AXBatchContext, actions []axBatchAction, start int) error { + if context == nil || s.noTreeWindowEntry(context.WindowRef) == nil { + return nil } - return nil -} - -// axNoTreeWindowBatch runs a batch on a window without an accessibility -// tree. Only activate, close and window waits can work there; anything -// else is rejected before the first action, so no input is sent. -func (s *Service) axNoTreeWindowBatch( - actions []axBatchAction, - window *AXWindowRefEntry, - notes []string, - requested int, -) ToolCallResult { for index, action := range actions { switch action.actionType { case "activate", "close": - if action.targetRef == "" || s.noTreeWindowEntry(action.targetRef) != nil { - continue - } - return axArgErrorResult(axInvalid(fmt.Sprintf("actions[%d]", index), - "one call cannot act on a window without an accessibility tree "+ - "and on other windows; split it into separate calls", - `{"mode":"act","actions":[{"type":"activate","ref":"`+window.Ref+`"}]}`), requested) + continue case "wait": if action.wait != nil && action.wait.window != "" { continue } } - return axNoTreeWindowResult(&axNoTreeWindowError{ref: window.Ref}, - fmt.Sprintf("actions[%d]", index), requested) - } - lines := append([]string{}, notes...) - emitted := false - closed := map[string]bool{} - for index, action := range actions { - if action.actionType == "wait" { - result := s.runAXWait(&AXBatchContext{WindowRef: window.Ref}, action.wait) - if !axPostconditionPassed(result) { - return axNoTreeBatchStop(result, index, requested, emitted, lines) - } - lines = append(lines, firstLine(ResultText(result))) - continue - } - target := window - if action.targetRef != "" { - target = s.noTreeWindowEntry(action.targetRef) + return &axArgError{ + Code: contract.AXCodeAppUnavailable, + path: fmt.Sprintf("actions[%d]", start+index), + Message: (&axNoTreeWindowError{ref: context.WindowRef}).Error(), + Hint: GUIScreenshotCall, } - result := s.axNoTreeWindowCommand(action.actionType, target) - if result.IsError { - return axNoTreeBatchStop(result, index, requested, emitted, lines) - } - emitted = true - verb := "activated" - if action.actionType == "close" { - verb = "closed" - closed[target.Ref] = true - } - lines = append(lines, verb+" "+target.Ref) } - if !closed[window.Ref] { - lines = append(lines, window.Ref+" "+axNoTreeWindowMark) - } - result := TextResult(strings.Join(lines, "\n"), false) + return nil +} + +func (s *Service) axNoTreeBatchAction(action string, window *AXWindowRefEntry) ToolCallResult { + result := s.axNoTreeWindowCommand(action, window) + if result.IsError { + return result + } + result = axNoTreeBatchObservation(window) + verb := "activated" + if action == "close" { + verb = "closed" + } + result.Content = append([]ContentItem{{Type: "text", Text: verb + " " + window.Ref}}, result.Content...) + result.StructuredContent["input_events_emitted"] = true + return result +} + +func axNoTreeBatchObservation(window *AXWindowRefEntry) ToolCallResult { + result := TextResult(window.Ref+" "+axNoTreeWindowMark, false) result.StructuredContent = map[string]any{ - "input_events_emitted": emitted, + "input_events_emitted": false, "final_observation_returned": false, "final_observation_kind": "none", "observation_required": false, "evidence_scope": "delivery_or_local_ui", - "a11y_batch_result": map[string]any{ - "actions_requested": requested, - "actions_completed_without_call_error": len(actions), - }, } return result } +func (s *Service) axNoTreeBatchWindowContinuity(context *AXBatchContext) (axWindowContinuity, bool) { + if s == nil || context == nil || s.noTreeWindowEntry(context.WindowRef) == nil { + return axWindowUnknown, false + } + if !validInternalWindowHandle(context.WindowHandle) { + return axWindowUnknown, true + } + inventory, _ := s.listDesktopWindows(false) + if inventory == nil { + return axWindowUnknown, true + } + for _, window := range inventory.windows { + if window.handle == context.WindowHandle { + return axWindowSame, true + } + } + return axWindowChanged, true +} + // axNoTreeWindowCommand asks the provider to activate or close the native // window and to confirm the effect with the window manager. func (s *Service) axNoTreeWindowCommand(action string, window *AXWindowRefEntry) ToolCallResult { @@ -182,32 +164,6 @@ func (s *Service) axNoTreeWindowCommand(action string, window *AXWindowRefEntry) return result } -func axNoTreeBatchStop( - result ToolCallResult, - index, requested int, - emitted bool, - lines []string, -) ToolCallResult { - if result.StructuredContent == nil { - result.StructuredContent = map[string]any{} - } - if emitted { - result.StructuredContent["input_events_emitted"] = true - } - result.StructuredContent["a11y_batch_result"] = map[string]any{ - "actions_requested": requested, - "actions_completed_without_call_error": index, - "failed_action_index": index, - } - if len(lines) > 0 { - result.Content = append( - []ContentItem{{Type: "text", Text: strings.Join(lines, "\n")}}, - result.Content..., - ) - } - return result -} - // asNoTreeWindowError reports whether err is a read of a window without a tree. func asNoTreeWindowError(err error) (*axNoTreeWindowError, bool) { var noTree *axNoTreeWindowError diff --git a/computer-use/internal/core/ax_native_window_batch_test.go b/computer-use/internal/core/ax_native_window_batch_test.go new file mode 100644 index 0000000..dd34fac --- /dev/null +++ b/computer-use/internal/core/ax_native_window_batch_test.go @@ -0,0 +1,321 @@ +package core + +import ( + "errors" + "fmt" + "reflect" + "slices" + "strings" + "testing" + + "github.com/microsoft/Hybrid-Computer-Use/computer-use/internal/contract" +) + +type axNoTreeActivationBackend struct { + *axActivationBackend + listings int + issues []ProviderInventoryIssue + inventoryError error + nativeReads int +} + +func (backend *axNoTreeActivationBackend) ObserveApps(request ProviderObserveAppsRequest) (ProviderAccessibilityResult, error) { + backend.listings++ + if backend.inventoryError != nil { + return ProviderAccessibilityResult{}, backend.inventoryError + } + result, err := backend.axActivationBackend.ObserveApps(request) + result.InventoryIssues = backend.issues + for appIndex := range result.Apps { + for windowIndex := range result.Apps[appIndex].Windows { + window := &result.Apps[appIndex].Windows[windowIndex] + window.NoAccessibilityTree = window.Handle == reportMainHandle + } + } + return result, err +} + +func (backend *axNoTreeActivationBackend) Observe(request ProviderObserveRequest) (ProviderAccessibilityResult, error) { + if request.WindowHandle == reportMainHandle { + backend.nativeReads++ + return ProviderAccessibilityResult{}, fmt.Errorf("native window has no accessibility tree") + } + return backend.axActivationBackend.Observe(request) +} + +func axNoTreeActivationService(t *testing.T) (*Service, *axNoTreeActivationBackend, axActivationRefs, axActivationRefs) { + t.Helper() + service, activation, native, tree := axActivationService(t, true) + backend := &axNoTreeActivationBackend{axActivationBackend: activation} + service.Backend = backend + if listed := service.listApps(); listed.IsError { + t.Fatalf("list native window: %s", ResultText(listed)) + } + return service, backend, native, tree +} + +func TestAXNoTreeMixedActivationToTree(t *testing.T) { + for _, byTitle := range []bool{false, true} { + t.Run(fmt.Sprintf("title_%t", byTitle), func(t *testing.T) { + service, backend, native, tree := axNoTreeActivationService(t) + activation := map[string]any{"type": "activate", "ref": tree.window} + if byTitle { + activation = map[string]any{"type": "activate", "title": "Notes"} + } + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "window_ref": native.window, "observe": "tree", "actions": []any{ + map[string]any{"type": "activate", "ref": native.window}, + activation, + map[string]any{"type": "type", "text": "untargeted"}, + map[string]any{"type": "type", "ref": tree.entry, "text": "targeted"}, + }, + }) + if result.IsError { + t.Fatalf("native -> tree rejected: %s", ResultText(result)) + } + want := []string{"activate@" + reportMainHandle, "activate@" + reportOtherHandle, "type@" + reportOtherHandle, "type@" + reportOtherHandle} + if !slices.Equal(backend.deliveries, want) { + t.Errorf("deliveries = %v, want %v", backend.deliveries, want) + } + axActivationEvidence(t, result, native.window, tree.window, tree.window, tree.window) + if backend.nativeReads != 0 || result.StructuredContent["final_observation_returned"] != true || !strings.Contains(ResultText(result), `window: "Notes" [ref=`+tree.window+`]`) { + t.Errorf("wrong final observation: native reads %d, result %#v\n%s", backend.nativeReads, result.StructuredContent, ResultText(result)) + } + }) + } +} + +func TestAXNoTreeMixedActivationFinalNative(t *testing.T) { + service, backend, native, tree := axNoTreeActivationService(t) + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "window_ref": tree.window, "observe": "tree", "actions": []any{ + map[string]any{"type": "activate", "ref": tree.window}, + map[string]any{"type": "type", "ref": tree.entry, "text": "before switch"}, + map[string]any{"type": "activate", "title": "Report"}, + }, + }) + if result.IsError { + t.Fatalf("tree -> native rejected: %s", ResultText(result)) + } + want := []string{"activate@" + reportOtherHandle, "type@" + reportOtherHandle, "activate@" + reportMainHandle} + if !slices.Equal(backend.deliveries, want) { + t.Errorf("deliveries = %v, want %v", backend.deliveries, want) + } + axActivationEvidence(t, result, tree.window, tree.window, native.window) + if backend.nativeReads != 0 || !strings.Contains(ResultText(result), native.window+" "+axNoTreeWindowMark) { + t.Errorf("native final observation: reads %d, text %s", backend.nativeReads, ResultText(result)) + } + for key, want := range map[string]any{"input_events_emitted": true, "final_observation_returned": false, "final_observation_kind": "none", "observation_required": false} { + if result.StructuredContent[key] != want { + t.Errorf("%s = %v, want %v", key, result.StructuredContent[key], want) + } + } +} + +func TestAXNoTreeMixedInputRejectedBeforeAnyDelivery(t *testing.T) { + for _, action := range []map[string]any{ + {"type": "keypress", "keys": []any{"ENTER"}}, + {"type": "type", "text": "must not be sent"}, + {"type": "wait", "text": "not a window wait", "timeout_ms": 200}, + } { + t.Run(fmt.Sprint(action["type"]), func(t *testing.T) { + service, backend, native, tree := axNoTreeActivationService(t) + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "actions": []any{ + map[string]any{"type": "activate", "ref": tree.window}, + map[string]any{"type": "activate", "ref": native.window}, + action, + map[string]any{"type": "activate", "ref": tree.window}, + }, + }) + if !result.IsError || result.StructuredContent[contract.AXPublicErrorCodeKey] != contract.AXCodeAppUnavailable || result.StructuredContent[contract.AXPublicErrorPathKey] != "actions[2]" || !strings.Contains(ResultText(result), "no accessibility tree") { + t.Errorf("want native input preflight error, got %#v\n%s", result.StructuredContent, ResultText(result)) + } + if len(backend.deliveries) != 0 || axEvidenceReportsInputEmission(result.StructuredContent) { + t.Errorf("invalid batch emitted input: %v, %#v", backend.deliveries, result.StructuredContent) + } + }) + } +} + +func TestAXNoTreeSegmentValidation(t *testing.T) { + service, backend, native, tree := axNoTreeActivationService(t) + allowed := []axBatchAction{ + {actionType: "activate", targetRef: native.window}, + {actionType: "wait", wait: &axWait{window: "Report"}}, + {actionType: "close", targetRef: native.window}, + } + if err := service.validateAXNoTreeSegment(&AXBatchContext{WindowRef: native.window}, allowed, 4); err != nil { + t.Fatalf("native window operations rejected: %v", err) + } + for _, action := range []axBatchAction{ + {actionType: "keypress", key: "ENTER"}, + {actionType: "type", text: "unsent"}, + {actionType: "click", targetRef: tree.button}, + {actionType: "scroll", direction: "down"}, + {actionType: "set_value", targetRef: tree.entry, value: "unsent"}, + {actionType: "select", targetRef: tree.entry, toRef: tree.entry}, + {actionType: "wait"}, + {actionType: "wait", wait: &axWait{ref: tree.entry}}, + {actionType: "wait", wait: &axWait{query: map[string]any{"role": "button"}}}, + {actionType: "wait", wait: &axWait{settle: true}}, + } { + actions := []axBatchAction{allowed[0], action} + err := service.validateAXNoTreeSegment(&AXBatchContext{WindowRef: native.window}, actions, 4) + var argument *axArgError + if !errors.As(err, &argument) || argument.Code != contract.AXCodeAppUnavailable || argument.path != "actions[5]" || argument.Hint != GUIScreenshotCall { + t.Errorf("%s rejection = %v", action.actionType, err) + } + if err := service.validateAXNoTreeSegment(&AXBatchContext{WindowRef: tree.window}, actions, 4); err != nil { + t.Errorf("tree segment rejected by native validation: %v", err) + } + } + if len(backend.deliveries) != 0 || backend.nativeReads != 0 { + t.Errorf("segment validation reached input/tree: %v, %d", backend.deliveries, backend.nativeReads) + } +} + +func TestAXNoTreeBatchWindowContinuityUsesFreshStrongHandle(t *testing.T) { + for _, scenario := range []struct { + name string + mutate func(*axNoTreeActivationBackend) + want axWindowContinuity + }{ + {name: "same", want: axWindowSame}, + {name: "renamed", want: axWindowSame, mutate: func(backend *axNoTreeActivationBackend) { backend.windows[0].title = "Renamed" }}, + {name: "other_window_focused", want: axWindowSame, mutate: func(backend *axNoTreeActivationBackend) { backend.windows[1].active = true }}, + {name: "gone", want: axWindowChanged, mutate: func(backend *axNoTreeActivationBackend) { backend.windows = backend.windows[1:] }}, + {name: "same_title_new_handle", want: axWindowChanged, mutate: func(backend *axNoTreeActivationBackend) { + backend.owners[testHandleScene] = backend.owners[reportMainHandle] + backend.windows[0].handle = testHandleScene + }}, + {name: "incomplete_present", want: axWindowUnknown, mutate: func(backend *axNoTreeActivationBackend) { + backend.issues = []ProviderInventoryIssue{{Reason: "enumeration incomplete"}} + }}, + {name: "incomplete_absent", want: axWindowUnknown, mutate: func(backend *axNoTreeActivationBackend) { + backend.windows = backend.windows[1:] + backend.issues = []ProviderInventoryIssue{{Reason: "enumeration incomplete"}} + }}, + {name: "failed", want: axWindowUnknown, mutate: func(backend *axNoTreeActivationBackend) { backend.inventoryError = errors.New("inventory unavailable") }}, + } { + t.Run(scenario.name, func(t *testing.T) { + service, backend, native, _ := axNoTreeActivationService(t) + if scenario.mutate != nil { + scenario.mutate(backend) + } + before := backend.listings + state, handled := service.axNoTreeBatchWindowContinuity(&AXBatchContext{WindowRef: native.window, WindowHandle: reportMainHandle}) + if !handled || state != scenario.want || backend.listings != before+1 { + t.Errorf("continuity = %s/%t, listings %d -> %d; want %s and one fresh inventory", state, handled, before, backend.listings, scenario.want) + } + if backend.nativeReads != 0 || len(backend.deliveries) != 0 { + t.Errorf("continuity read native tree or sent input: %d, %v", backend.nativeReads, backend.deliveries) + } + }) + } + service, backend, native, tree := axNoTreeActivationService(t) + before := backend.listings + if _, handled := service.axNoTreeBatchWindowContinuity(&AXBatchContext{WindowRef: tree.window, WindowHandle: reportOtherHandle}); handled || backend.listings != before { + t.Errorf("ordinary tree continuity intercepted: %t, listings %d -> %d", handled, before, backend.listings) + } + state, handled := service.axNoTreeBatchWindowContinuity(&AXBatchContext{WindowRef: native.window}) + if !handled || state != axWindowUnknown || backend.listings != before { + t.Errorf("missing strong handle should stay unknown without enumeration: %s/%t", state, handled) + } +} + +func TestAXNoTreeBatchActionAndObservationEvidence(t *testing.T) { + for _, action := range []string{"activate", "close"} { + t.Run(action, func(t *testing.T) { + service, backend := windowScriptService(sceneApps()) + window := service.noTreeWindowEntry(noTreeWindowRef(t, service)) + backend.windowResponse = &ProviderWindowResult{OK: true} + result := service.axNoTreeBatchAction(action, window) + if result.IsError || !slices.Equal(backend.calls, []string{"window_" + action}) || !strings.Contains(ResultText(result), axNoTreeWindowMark) { + t.Fatalf("native action failed: %v, %s", backend.calls, ResultText(result)) + } + for key, want := range map[string]any{"input_events_emitted": true, "final_observation_returned": false, "final_observation_kind": "none", "observation_required": false} { + if result.StructuredContent[key] != want { + t.Errorf("%s = %v, want %v", key, result.StructuredContent[key], want) + } + } + observation := axNoTreeBatchObservation(window) + if observation.IsError || observation.StructuredContent["input_events_emitted"] != false || len(backend.calls) != 1 { + t.Errorf("native final observation emitted input: %#v, %v", observation, backend.calls) + } + }) + } +} + +func TestAXNoTreeBatchActionPreservesProviderFailure(t *testing.T) { + for _, emitted := range []bool{false, true} { + t.Run(fmt.Sprintf("emitted_%t", emitted), func(t *testing.T) { + service, backend := windowScriptService(sceneApps()) + window := service.noTreeWindowEntry(noTreeWindowRef(t, service)) + backend.windowResponse = &ProviderWindowResult{ + Error: "provider could not confirm activation", + Failure: &ProviderFailure{Code: "input_failed", Phase: "emit", FactsComplete: true}, + Mutation: &ProviderMutationFacts{InputEventsEmitted: emitted}, + } + want := service.axNoTreeWindowCommand("activate", window) + got := service.axNoTreeBatchAction("activate", window) + if !got.IsError || !reflect.DeepEqual(got, want) || got.StructuredContent["input_events_emitted"] != emitted { + t.Errorf("failure facts changed: got %#v, want %#v", got, want) + } + }) + } +} + +func TestAXNoTreeBatchKeepsNativeWindowBinding(t *testing.T) { + for _, wrongClose := range []bool{false, true} { + t.Run(fmt.Sprintf("wrong_close_%t", wrongClose), func(t *testing.T) { + apps := sceneApps() + nativeApp := &apps[len(apps)-1] + second := nativeApp.Windows[0] + second.Handle = testHandleNotes + second.Title = "Render - Paint3D" + nativeApp.Windows = append(nativeApp.Windows, second) + service, backend := windowScriptService(apps) + backend.windowResponse = &ProviderWindowResult{OK: true} + firstRef := noTreeWindowRef(t, service) + inventory, failure := service.listDesktopWindows(false) + if inventory == nil { + t.Fatalf("list native windows: %s", ResultText(failure)) + } + secondRef := "" + for _, window := range inventory.windows { + if window.handle == second.Handle { + secondRef = window.ref + } + } + if secondRef == "" { + t.Fatal("missing second native window") + } + closeRef := secondRef + if wrongClose { + closeRef = firstRef + } + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "window_ref": firstRef, "observe": "tree", "actions": []any{ + map[string]any{"type": "activate", "ref": firstRef}, + map[string]any{"type": "activate", "title": "Render"}, + map[string]any{"type": "wait", "window": "Render", "timeout_ms": 200}, + map[string]any{"type": "close", "ref": closeRef}, + }, + }) + if wrongClose { + if !result.IsError || len(backend.calls) != 0 || axEvidenceReportsInputEmission(result.StructuredContent) || result.StructuredContent[contract.AXPublicErrorCodeKey] != contract.AXCodeInvalidArgument { + t.Fatalf("close in previous native window was not rejected before input: %v, %#v", backend.calls, result) + } + return + } + if result.IsError || !slices.Equal(backend.calls, []string{"window_activate", "window_activate", "window_close"}) { + t.Fatalf("all native batch failed: %v, %s", backend.calls, ResultText(result)) + } + axActivationEvidence(t, result, firstRef, secondRef, secondRef, secondRef) + if result.StructuredContent["input_events_emitted"] != true || result.StructuredContent["final_observation_returned"] != false || result.StructuredContent["final_observation_kind"] != "none" || result.StructuredContent["observation_required"] != false { + t.Errorf("native final evidence changed: %#v", result.StructuredContent) + } + }) + } +} diff --git a/computer-use/internal/core/ax_projection.go b/computer-use/internal/core/ax_projection.go index 6c625a4..93b0557 100644 --- a/computer-use/internal/core/ax_projection.go +++ b/computer-use/internal/core/ax_projection.go @@ -539,10 +539,10 @@ func resolveProviderBoundObservationRoot( } if strings.TrimSpace(entry.WindowHandle) == "" || entry.WindowHandle != snapshot.WindowHandle { - return nil, errors.New( - "stale_window: root ref belongs to a different window; " + + return nil, &axRefLifecycleError{kind: "window", lifecycle: AXRefStale, + message: "stale_window: root ref belongs to a different window; " + "observe or find the target again", - ) + } } matches := make([]*ElementRecord, 0, 2) for index := range snapshot.Elements { @@ -565,17 +565,17 @@ func resolveProviderBoundObservationRoot( } switch len(matches) { case 0: - return nil, errors.New( - "stale_element_ref: the provider-bound root is no longer present; " + + return nil, &axRefLifecycleError{kind: "element", lifecycle: AXRefStale, + message: "stale_element_ref: the provider-bound root is no longer present; " + "use find to obtain a current ref", - ) + } case 1: return matches[0], nil default: - return nil, errors.New( - "ambiguous_target: the provider-bound root does not identify one " + + return nil, &axRefLifecycleError{kind: "element", lifecycle: AXRefStale, + message: "ambiguous_target: the provider-bound root does not identify one " + "current canonical element; use find to obtain a current ref", - ) + } } } diff --git a/computer-use/internal/core/ax_public_action_hint_test.go b/computer-use/internal/core/ax_public_action_hint_test.go new file mode 100644 index 0000000..d6eecc9 --- /dev/null +++ b/computer-use/internal/core/ax_public_action_hint_test.go @@ -0,0 +1,136 @@ +package core + +import ( + "reflect" + "strings" + "testing" + + "github.com/microsoft/Hybrid-Computer-Use/computer-use/internal/contract" +) + +func TestAXCallShapeHintsPreserveIntent(t *testing.T) { + for _, test := range []struct { + name string + input string + path string + corrected string + }{ + { + name: "activate mode keeps window and observation", + input: `{"mode":"activate","actions":[{"type":"activate","ref":"w7"}],"observe":"full"}`, + path: "mode", + corrected: `{"mode":"act","actions":[{"type":"activate","ref":"w7"}],"observe":"full"}`, + }, + { + name: "activate mode keeps title and later input", + input: `{"mode":"activate","window_ref":"w7","actions":[{"type":"activate","title":"Report"},{"type":"type","text":"Quarterly report"}],"observe":"tree"}`, + path: "mode", + corrected: `{"mode":"act","window_ref":"w7","actions":[{"type":"activate","title":"Report"},{"type":"type","text":"Quarterly report"}],"observe":"tree"}`, + }, + { + name: "activate mode also repairs window field", + input: `{"mode":"activate","actions":[{"type":"activate","window_ref":"w7"}],"observe":"none"}`, + path: "mode", + corrected: `{"mode":"act","actions":[{"type":"activate","ref":"w7"}],"observe":"none"}`, + }, + { + name: "settle keeps preceding input timeout and following input", + input: `{"mode":"act","window_ref":"w7","actions":[{"type":"keypress","keys":["CTRL","O"]},{"type":"settle","timeout_ms":1200},{"type":"type","text":"report.txt"}],"observe":"tree"}`, + path: "actions[1].type", + corrected: `{"mode":"act","window_ref":"w7","actions":[{"type":"keypress","keys":["CTRL","O"]},{"type":"wait","settle":true,"timeout_ms":1200},{"type":"type","text":"report.txt"}],"observe":"tree"}`, + }, + { + name: "settle removes unsupported fields", + input: `{"mode":"act","actions":[{"type":"click","ref":"r9"},{"type":"settle","direction":"down","timeout_ms":1500}],"observe":"none"}`, + path: "actions[1].type", + corrected: `{"mode":"act","actions":[{"type":"click","ref":"r9"},{"type":"wait","settle":true,"timeout_ms":1500}],"observe":"none"}`, + }, + { + name: "activate mode and settle repair together", + input: `{"mode":"activate","actions":[{"type":"activate","ref":"w7"},{"type":"settle","timeout_ms":1500}],"observe":"full"}`, + path: "mode", + corrected: `{"mode":"act","actions":[{"type":"activate","ref":"w7"},{"type":"wait","settle":true,"timeout_ms":1500}],"observe":"full"}`, + }, + { + name: "double click suggests one invocation on the same ref", + input: `{"mode":"act","actions":[{"type":"double_click","ref":"r9"}],"observe":"full"}`, + path: "actions[0].type", + corrected: `{"mode":"act","actions":[{"type":"click","ref":"r9"}],"observe":"full"}`, + }, + } { + t.Run(test.name, func(t *testing.T) { + args := decodeTestArguments(t, test.input) + original := axCloneValue(args) + request, err := DecodeAXRequest(args) + if request != nil || err == nil || err.Code != contract.AXCodeInvalidArgument || err.path != test.path { + t.Fatalf("want rejection at %s, got request=%#v error=%#v", test.path, request, err) + } + if !reflect.DeepEqual(args, original) { + t.Fatal("hint generation changed the rejected call") + } + if hintErr := axHintDecodeError(args, err); hintErr != nil { + t.Fatalf("hint %s does not decode: %v", err.Hint, hintErr) + } + if len(err.Hint) > contract.MaxArgumentHintBytes { + t.Fatalf("hint exceeds limit: %s", err.Hint) + } + corrected, ok := axSpliceHint(args, err.path, err.Hint) + if !ok || !reflect.DeepEqual(corrected, decodeTestArguments(t, test.corrected)) { + t.Fatalf("hint %s produces %s; want %s", err.Hint, contract.CompactJSON(corrected), test.corrected) + } + if schemaErr := contract.ValidateToolArguments(contract.PublicToolAX, corrected); schemaErr != nil { + t.Fatalf("corrected call fails public schema: %v", schemaErr) + } + }) + } +} + +func TestAXDoubleClickHintExplainsGUICallShape(t *testing.T) { + for _, action := range []string{ + `{"type":"double_click","ref":"r9"}`, + `{"type":"double_click","x":320,"y":240}`, + } { + args := decodeTestArguments(t, `{"mode":"act","actions":[`+action+`]}`) + _, err := DecodeAXRequest(args) + if err == nil || err.Code != contract.AXCodeInvalidArgument { + t.Fatalf("double_click must remain rejected: %#v", err) + } + for _, part := range []string{"AX click", "once", "GUI", "actions", `type "double_click"`, "x/y", "full-desktop screenshot"} { + if !strings.Contains(err.Message, part) { + t.Errorf("message %q does not explain %q", err.Message, part) + } + } + if hintErr := axHintDecodeError(args, err); hintErr != nil { + t.Fatalf("hint %s does not decode: %v", err.Hint, hintErr) + } + } + if err := contract.ValidateToolArguments(contract.PublicToolGUI, decodeTestArguments(t, + `{"actions":[{"type":"double_click","x":320,"y":240}]}`)); err != nil { + t.Fatalf("suggested GUI call shape is unsupported: %v", err) + } +} + +func TestAXCallShapeHintsRemainValidWithMissingContext(t *testing.T) { + for _, raw := range []string{ + `{"mode":"act","actions":[{"type":"settle","timeout_ms":1200}]}`, + `{"mode":"act","actions":[{"type":"wait","text":"Ready"},{"type":"settle"}]}`, + `{"mode":"act","actions":[{"type":"click","ref":"r9"},{"type":"settle","gone":true,"timeout_ms":"1500"}]}`, + `{"mode":"activate","actions":[{"type":"activate","title":"` + strings.Repeat("long title ", 40) + `"}]}`, + } { + args := decodeTestArguments(t, raw) + original := axCloneValue(args) + request, err := DecodeAXRequest(args) + if request != nil || err == nil || err.Code != contract.AXCodeInvalidArgument { + t.Fatalf("call must remain rejected: request=%#v error=%#v", request, err) + } + if err.Hint == "" || len(err.Hint) > contract.MaxArgumentHintBytes { + t.Fatalf("hint is empty or exceeds limit: %q", err.Hint) + } + if hintErr := axHintDecodeError(args, err); hintErr != nil { + t.Fatalf("hint %s does not decode: %v", err.Hint, hintErr) + } + if !reflect.DeepEqual(args, original) { + t.Fatal("hint generation changed the rejected call") + } + } +} diff --git a/computer-use/internal/core/ax_public_request.go b/computer-use/internal/core/ax_public_request.go index cd20dad..3b82fda 100644 --- a/computer-use/internal/core/ax_public_request.go +++ b/computer-use/internal/core/ax_public_request.go @@ -25,6 +25,7 @@ var axPublicErrorCodes = []string{ contract.AXCodeInvalidArgument, contract.AXCodeStaleRef, contract.AXCodeWindowChanged, contract.AXCodeWaitTimeout, contract.AXCodeEffectUnknown, contract.AXCodeAppUnavailable, contract.AXCodeRuntimeError, contract.AXCodeAmbiguousWindow, contract.AXCodeWindowNotFound, contract.AXCodeNoFocusedWindow, + contract.AXCodeGlobalInputForbidden, } // axArgError is a rejected argument. It is returned before any input. @@ -91,6 +92,7 @@ type axRequest struct { limit int cursor string actions []axBatchAction + readText *ProviderTextRead } var ( @@ -102,19 +104,19 @@ var ( var axModeFields = map[string][]string{ contract.AXModeApps: {"mode"}, - contract.AXModeObserve: {"mode", "app", "window_ref", "scope", "observe", "root"}, + contract.AXModeObserve: {"mode", "app", "window_ref", "scope", "observe", "root", "read_text"}, contract.AXModeFind: {"mode", "app", "window_ref", "scope", "query", "limit", "cursor"}, contract.AXModeAct: {"mode", "window_ref", "actions", "observe"}, } // axActionFields lists the fields each action type accepts besides type. var axActionFields = map[string][]string{ - "click": {"ref"}, + "click": {"ref", "allow_global_input"}, "type": {"ref", "text"}, "keypress": {"ref", "keys"}, "set_value": {"ref", "value"}, "select": {"ref", "to_ref", "text"}, - "scroll": {"ref", "direction", "pages"}, + "scroll": {"ref", "direction", "pages", "allow_global_input"}, "activate": {"ref", "title"}, "close": {"ref", "title"}, "wait": {"window", "query", "ref", "value", "state", "text", "gone", "settle", "timeout_ms"}, @@ -166,6 +168,8 @@ func decodeAXCall(args map[string]any) (*axRequest, *axArgError) { contract.CompactJSON(rawMode), `{"mode":"apps"}`) if lower := strings.ToLower(mode); lower != mode && axModeFields[lower] != nil { err.Hint, err.fix = "", []axEdit{axSet("mode", lower)} + } else if mode == "activate" && args["actions"] != nil { + err.Hint, err.fix = "", []axEdit{axSet("mode", contract.AXModeAct)} } return nil, err } @@ -205,11 +209,24 @@ func decodeAXCall(args map[string]any) (*axRequest, *axArgError) { } switch mode { case contract.AXModeObserve: + if raw, present := args["read_text"]; present { + if request.readText, err = decodeAXTextRead(raw); err != nil { + return nil, err + } + if !AXElementRefPattern.MatchString(request.root) { + return nil, axInvalid("root", "read_text requires an element root rN", "").withRule(axRuleRequiredForMode).withFix(axSet("root", "r1")) + } + for _, field := range []string{"app", "scope", "observe"} { + if _, present := args[field]; present { + return nil, axInvalid(field, "read_text returns only the requested text page; omit "+field, "").withRule(axRuleFieldForMode).withFix(axRemove(field)) + } + } + } if request.observe == contract.ObserveNone { return nil, axInvalid("observe", `observe "none" returns nothing; to wait without input, send act with only wait steps`, "").withRule(axRuleObserveNone).withFix(axRemove("observe")) } - if request.app == "" && request.windowRef == "" { + if request.app == "" && request.windowRef == "" && request.readText == nil { return nil, axInvalid("app", "observe needs app or window_ref", "").withRule(axRuleRequiredForMode).withFix(axSet("window_ref", "w1")) } @@ -378,7 +395,18 @@ func decodeAXAction(object map[string]any, path string) (axBatchAction, *axArgEr } else { message = "action needs type: " + strings.Join(contract.AXActionTypes, ", ") } - return action, axInvalid(path+".type", message, axActionShapeHint(object)) + err := axInvalid(path+".type", message, axActionShapeHint(object)) + switch { + case actionType == "settle": + err.Message += `; use type "wait" with settle:true after an input step in the same call` + err.Hint = "" + err.fix = []axEdit{axSet(path+".type", "wait"), axSet(path+".settle", true)} + case actionType == "double_click": + err.Message += `; AX click invokes the element once; for a double click, call GUI with actions containing type "double_click" and x/y from a fresh full-desktop screenshot` + err.Hint = "" + err.fix = []axEdit{axSet(path+".type", "click")} + } + return action, err } action.actionType = actionType aliased, aliasErr := axNormalizeTextAlias(object, actionType, path) @@ -400,6 +428,13 @@ func decodeAXAction(object map[string]any, path string) (axBatchAction, *axArgEr } return action, err.withRule(axRuleFieldForType) } + if raw, present := object["allow_global_input"]; present { + allowed, ok := raw.(bool) + if !ok { + return action, axInvalid(path+".allow_global_input", "allow_global_input must be a boolean", "") + } + action.allowGlobalInput = &allowed + } if actionType == "wait" { wait, err := decodeAXWait(object, path) if err != nil { diff --git a/computer-use/internal/core/ax_ref_snapshot_assignment.go b/computer-use/internal/core/ax_ref_snapshot_assignment.go index 87c1a72..4acdc7e 100644 --- a/computer-use/internal/core/ax_ref_snapshot_assignment.go +++ b/computer-use/internal/core/ax_ref_snapshot_assignment.go @@ -121,6 +121,10 @@ func axNodeRefEligibility( } if !axNodeHasInteractionSemantics(node) { + if axNodeProviderIdentityComplete(node) && + (node.Properties.Value != "" || len(node.Properties.TextTruncated) > 0) { + return true, "", AXLocatorProviderBound + } return false, "interaction_semantics_unknown", "" } if axNodeIsUnnamedEditor(node) { diff --git a/computer-use/internal/core/ax_ref_snapshot_assignment_test.go b/computer-use/internal/core/ax_ref_snapshot_assignment_test.go new file mode 100644 index 0000000..944f63a --- /dev/null +++ b/computer-use/internal/core/ax_ref_snapshot_assignment_test.go @@ -0,0 +1,121 @@ +package core + +import ( + "encoding/json" + "os" + "strings" + "testing" + + "github.com/microsoft/Hybrid-Computer-Use/computer-use/internal/contract" +) + +func modalRefSnapshot(revision string, records ...ElementRecord) *AppSnapshot { + for index := range records { + records[index].Index = index + 1 + records[index].ProviderBusName = ":1.42" + records[index].ProcessGeneration = "epoch1" + records[index].ProviderWindowBusName = ":1.42" + records[index].ProviderWindowObjectPath = "/window" + records[index].ProviderWindowProcessGeneration = "epoch1" + } + return &AppSnapshot{ + App: AppDescriptor{Name: "Messages"}, + WindowHandle: "w2:" + strings.Repeat("a", 32), + SnapshotID: revision, Scope: contract.AXScopeWindow, + publicAppRef: "a1", publicWindowRef: "w1", Elements: records, + } +} + +func assignModalRefSnapshot(t *testing.T, service *Service, source *AppSnapshot) *AXSnapshot { + t.Helper() + tree, err := buildAXSnapshot(source) + if err != nil { + t.Fatal(err) + } + service.RefRegistry.AssignSnapshotRefs(tree) + source.AXSnapshot = tree + service.rememberSnapshot("messages", source) + return tree +} + +func TestModalScopePreservesRefUntilCompleteRead(t *testing.T) { + payload, err := os.ReadFile("../../testdata/provider-wire/modal-scope.response.json") + if err != nil { + t.Fatal(err) + } + var fixture struct { + Accessibility struct { + Snapshot AppSnapshot `json:"snapshot"` + } `json:"accessibility"` + } + if err := json.Unmarshal(payload, &fixture); err != nil { + t.Fatal(err) + } + for _, outcome := range []string{"same_path", "moved", "renamed", "ambiguous", "absent", "window_replaced"} { + t.Run(outcome, func(t *testing.T) { + service := NewService() + button := ElementRecord{ + RuntimeID: []int{0, 1}, ControlType: "push button", Name: "Invoice", + ProviderObjectPath: "/old/invoice", Actions: []string{"click"}, + } + before := assignModalRefSnapshot(t, service, modalRefSnapshot("before", button)) + ref := before.ByIndex[1].Ref + entry, err := service.RefRegistry.Resolve(ref, "messages") + if err != nil || entry.LocatorKind != AXLocatorSemantic { + t.Fatalf("initial semantic ref: %v, %#v", err, entry) + } + modal := modalRefSnapshot("modal", ElementRecord{ + RuntimeID: []int{0, 0}, ControlType: "push button", Name: "Close", + ProviderObjectPath: "/dialog/close", Actions: []string{"click"}, + }) + modal.Annotations = fixture.Accessibility.Snapshot.Annotations + current := assignModalRefSnapshot(t, service, modal) + if current.RefInventoryComplete || current.AcquisitionIncomplete || !current.IdentityCoverageIncomplete { + t.Fatalf("modal is a coverage boundary, not a failed read: %#v", current) + } + entry, err = service.RefRegistry.Resolve(ref, "messages") + if err != nil || entry.LastSeenRevision != "before" { + t.Fatalf("modal retired or refreshed the hidden ref: %v, %#v", err, entry) + } + target, err := semanticTargetFromRefEntry(entry) + if err != nil { + t.Fatal(err) + } + service.authorizeProviderRebind("messages", entry, target) + if target.AllowProviderRebind { + t.Fatal("hidden target allowed semantic rebind inside modal scope") + } + button.ProviderObjectPath = "/rebuilt/invoice" + after := modalRefSnapshot("after", button) + switch outcome { + case "moved": + after.Elements[0].RuntimeID = []int{0, 2} + case "renamed": + after.Elements[0].Name = "Delete" + case "ambiguous": + peer := after.Elements[0] + peer.Index, peer.RuntimeID, peer.ProviderObjectPath = 2, []int{0, 2}, "/peer/invoice" + after.Elements = append(after.Elements, peer) + case "absent": + after.Elements = nil + case "window_replaced": + after.Elements[0].ProviderWindowObjectPath = "/new/window" + } + current = assignModalRefSnapshot(t, service, after) + entry, err = service.RefRegistry.Resolve(ref, "messages") + if outcome != "same_path" { + if err == nil { + t.Fatalf("%s revived the old ref", outcome) + } + return + } + if err != nil || current.ByIndex[1].Ref != ref || entry.ProviderObjectPath != "/old/invoice" { + t.Fatalf("same-path rebuild lost ref or skipped live validation: %v, %#v", err, entry) + } + service.authorizeProviderRebind("messages", entry, target) + if !target.AllowProviderRebind { + t.Fatal("complete fresh snapshot did not allow guarded live rebind") + } + }) + } +} diff --git a/computer-use/internal/core/ax_renderer.go b/computer-use/internal/core/ax_renderer.go index 81db4e1..fc202bf 100644 --- a/computer-use/internal/core/ax_renderer.go +++ b/computer-use/internal/core/ax_renderer.go @@ -166,7 +166,7 @@ func axInlineCellValue(node *AXNode) (string, bool) { return "", false } value := strings.TrimSpace(strings.ToValidUTF8(node.Properties.Value, "\uFFFD")) - if comparableTreeText(value) == "" || semanticTextEqual(value, node.Name) || + if comparableTreeText(value) == "" || axValueRepeatsName(node) || pureNonSemanticUnicode(value, node) { return "", false } @@ -175,9 +175,21 @@ func axInlineCellValue(node *AXNode) (string, bool) { func renderAXProperties(node *AXNode, options AXRenderOptions) [][2]string { properties := make([][2]string, 0, 8) + valuePreview := node.Properties.Value + if !roleExposesWorkingValue(node.Role) { + if node.Role == "cell" || node.Role == "gridcell" { + valuePreview, _ = axInlineCellValue(node) + } else { + valuePreview, _ = shortenAtWord(valuePreview, maxFindTextRunes) + if valuePreview != "" && !axValueRepeatsName(node) && + !pureNonSemanticUnicode(node.Properties.Value, node) { + properties = append(properties, [2]string{"/text", valuePreview}) + } + } + } if node.Properties.Value != "" && (roleExposesWorkingValue(node.Role) && - !semanticTextEqual(node.Properties.Value, node.Name) && + !axValueRepeatsName(node) && !pureNonSemanticUnicode(node.Properties.Value, node)) { properties = append(properties, [2]string{"/value", node.Properties.Value}) } @@ -190,6 +202,22 @@ func renderAXProperties(node *AXNode, options AXRenderOptions) [][2]string { !semanticTextEqual(node.Properties.Description, node.Properties.Value)) { properties = append(properties, [2]string{"/description", node.Properties.Description}) } + for _, field := range []struct{ name, preview string }{ + {"name", node.Name}, {"value", valuePreview}, {"description", node.Properties.Description}, + } { + if field.name == "value" && axValueRepeatsName(node) { + continue + } + total := node.Properties.TextLengths[field.name] + if field.preview == "" || !axTextPreviewTruncated(node, field.name, field.preview) { + continue + } + hint := "use find to obtain a ref, then observe with root and read_text.field=" + field.name + if node.Ref != "" { + hint = contract.CompactJSON(axTextReadCall("", node.Ref, field.name, 0, contract.AXMaxReadText)) + } + properties = append(properties, [2]string{"/text-truncated", fmt.Sprintf("… %s length=%d codepoints; Linux: %s", field.name, total, hint)}) + } if node.Capabilities.SetValue { properties = append(properties, [2]string{"/set-value", "true"}) } @@ -221,6 +249,14 @@ func semanticTextEqual(left, right string) bool { comparableTreeText(left) == comparableTreeText(right) } +func axValueRepeatsName(node *AXNode) bool { + if semanticTextEqual(node.Properties.Value, node.Name) { + return true + } + record := node.SourceRecord + return record != nil && semanticTextEqual(record.Value, record.Name) +} + func pureNonSemanticUnicode(value string, node *AXNode) bool { if node == nil || node.Name != "" || node.Properties.Description != "" || diff --git a/computer-use/internal/core/ax_renderer_test.go b/computer-use/internal/core/ax_renderer_test.go index 03ae68b..772253b 100644 --- a/computer-use/internal/core/ax_renderer_test.go +++ b/computer-use/internal/core/ax_renderer_test.go @@ -7,6 +7,53 @@ import ( "github.com/microsoft/Hybrid-Computer-Use/computer-use/internal/contract" ) +func TestObserveShowsTextInterfaceOnlyParagraph(t *testing.T) { + snapshot := cellValueSnapshot() + snapshot.Elements[2].ControlType = "paragraph" + snapshot.Elements[2].Name = "" + snapshot.Elements[2].Value = "The paragraph body comes from Text, not the accessible name." + text := typedTreeTextForTest(t, snapshot) + if !strings.Contains(text, snapshot.Elements[2].Value) { + t.Fatalf("Text-only body missing: %s", text) + } +} + +func TestClippedCellTextReportsLengthAndReadRoute(t *testing.T) { + snapshot := cellValueSnapshot() + text := typedTreeTextForTest(t, snapshot) + for _, want := range []string{"/text-truncated:", "value length=", "read_text"} { + if !strings.Contains(text, want) { + t.Fatalf("missing %q: %s", want, text) + } + } + if strings.Contains(text, snapshot.Elements[3].Value) { + t.Fatal("the ordinary tree must not include the full long value") + } +} + +func TestFindDoesNotRepeatTruncationForIdenticalNameAndValue(t *testing.T) { + snapshot := cellValueSnapshot() + record := &snapshot.Elements[2] + record.ControlType = "label" + record.Name = strings.Repeat("x", 500) + "..." + record.Value = record.Name + record.TextLengths = map[string]int{"name": 501, "value": 501} + record.TextTruncated = []string{"name", "value"} + typed, err := buildAXSnapshot(snapshot) + if err != nil { + t.Fatal(err) + } + text := FindMatchUnit(typed, *record, 1) + if strings.Count(text, "/text-truncated:") != 1 || !strings.Contains(text, "name length=501") { + t.Fatalf("find duplicated the truncation hint after shortening the name: %s", text) + } + for _, property := range renderAXProperties(findDisplayNode(typed.ByIndex[record.Index]), AXRenderOptions{}) { + if property[0] == "/text" || property[0] == "/value" { + t.Fatalf("find context duplicated a value that equals the canonical name: %v", property) + } + } +} + func TestStructuredAXYAMLSupportsValueAndChildren(t *testing.T) { root := &AXNode{ ID: "0", diff --git a/computer-use/internal/core/ax_request.go b/computer-use/internal/core/ax_request.go index cb207c1..d627c07 100644 --- a/computer-use/internal/core/ax_request.go +++ b/computer-use/internal/core/ax_request.go @@ -126,6 +126,9 @@ func (s *Service) axDecodedRequest(request *axRequest) ToolCallResult { contract.AXModeApps, ) case contract.AXModeObserve: + if request.readText != nil { + return WithAXModeEvidence(s.axReadText(request), contract.AXModeObserve) + } return s.axObserve(request, obs) case contract.AXModeFind: return s.axFind(request) diff --git a/computer-use/internal/core/ax_request_test.go b/computer-use/internal/core/ax_request_test.go index 32a345e..f51cff5 100644 --- a/computer-use/internal/core/ax_request_test.go +++ b/computer-use/internal/core/ax_request_test.go @@ -7,6 +7,38 @@ import ( "github.com/microsoft/Hybrid-Computer-Use/computer-use/internal/contract" ) +func TestDecodeAXGlobalInputPolicy(t *testing.T) { + for _, actionType := range []string{"click", "scroll"} { + for _, policy := range []any{nil, false, true} { + t.Run(actionType+"/"+contract.CompactJSON(policy), func(t *testing.T) { + action := map[string]any{"type": actionType, "ref": "r1"} + if actionType == "scroll" { + action["direction"] = "down" + } + if policy != nil { + action["allow_global_input"] = policy + } + args := map[string]any{"mode": "act", "actions": []any{action}} + if err := contract.ValidateToolArguments(contract.PublicToolAX, args); err != nil { + t.Fatalf("schema rejected policy: %v", err) + } + request, err := DecodeAXRequest(args) + if err != nil { + t.Fatal(err) + } + got := request.actions[0].allowGlobalInput + if policy == nil { + if got != nil { + t.Fatalf("omitted policy became explicit: %v", *got) + } + } else if got == nil || *got != policy.(bool) { + t.Fatalf("lost explicit policy %v: %v", policy, got) + } + }) + } + } +} + func TestDecodeAXRequestAcceptsEachMode(t *testing.T) { for name, args := range map[string]map[string]any{ "apps": {"mode": "apps"}, diff --git a/computer-use/internal/core/ax_response_budget.go b/computer-use/internal/core/ax_response_budget.go index 225c716..0a7af1b 100644 --- a/computer-use/internal/core/ax_response_budget.go +++ b/computer-use/internal/core/ax_response_budget.go @@ -96,6 +96,9 @@ func EnforceAXResponseBudget(result ToolCallResult) ToolCallResult { limit := MaxInlineChars() const evidenceReserve = 2048 alreadyTruncated := resultContainsInlineTruncation(result) + if result.StructuredContent["text_page"] != nil { + alreadyTruncated = false + } if result.StructuredContent["find_delivery_complete"] == true { // Application text is not truncation evidence for atomic find units. alreadyTruncated = result.StructuredContent["truncation_reason"] == "response_budget" @@ -157,7 +160,7 @@ func EnforceAXResponseBudget(result ToolCallResult) ToolCallResult { reason = "structural_budget" } treeComplete := !truncated && !structuralBudgetApplied && - coverageIncomplete == 0 + coverageIncomplete == 0 && result.StructuredContent["text_page"] == nil responseChars := textContentChars(result) if result.StructuredContent != nil { result.StructuredContent["response_chars"] = responseChars @@ -197,7 +200,7 @@ func providerCoverageSummary( switch reason { case "filtered", "pressure": hasProjection = true - case "collapsed_menu", "provider_children_not_rendered": + case "collapsed_menu", "provider_children_not_rendered", "modal_dialog": hasExpandable = true case "": hasAcquisitionGap = true diff --git a/computer-use/internal/core/ax_runtime_failure_test.go b/computer-use/internal/core/ax_runtime_failure_test.go new file mode 100644 index 0000000..89119ac --- /dev/null +++ b/computer-use/internal/core/ax_runtime_failure_test.go @@ -0,0 +1,184 @@ +package core + +import ( + "bytes" + "encoding/base64" + "fmt" + "image" + "image/color" + "image/png" + "strings" + "testing" + + "github.com/microsoft/Hybrid-Computer-Use/computer-use/internal/contract" +) + +type unconfirmedSetValueBackend struct { + *reportBackend + failure ProviderAccessibilityResult + failAt int + snapshotInFailure bool +} + +func (*unconfirmedSetValueBackend) HostInfo() HostBackendInfo { + return HostBackendInfo{Platform: "linux"} +} + +func (backend *unconfirmedSetValueBackend) ResolveAndAct(request ProviderActionRequest) (ProviderAccessibilityResult, error) { + response, err := backend.reportBackend.ResolveAndAct(request) + if len(backend.actionRequests) != backend.failAt { + return response, err + } + failure := backend.failure + if backend.snapshotInFailure { + failure.Snapshot = response.Snapshot + } + return failure, nil +} + +func TestSetValueUnconfirmedEditorStopsBatchDespiteObservedChanges(t *testing.T) { + for _, scenario := range []struct { + name string + focusChanged bool + pixelsChanged bool + snapshotInFailure bool + }{ + {name: "unchanged"}, + {name: "focus_changed", focusChanged: true}, + {name: "editor_and_pixels_changed", focusChanged: true, pixelsChanged: true}, + {name: "changed_failure_snapshot", focusChanged: true, pixelsChanged: true, snapshotInFailure: true}, + } { + for _, failAt := range []int{1, 2} { + for _, observe := range []string{contract.ObserveNone, contract.ObserveFull} { + t.Run(fmt.Sprintf("%s/action_%d/%s", scenario.name, failAt, observe), func(t *testing.T) { + service, desktop := reportService(t, reportMain()) + window, refs := reportObserve(t, service, "Size", "Interface") + before := desktop.snapshot(reportMain()) + desktop.desktopResponse = testDesktopFrame(t, "before", 40, 30) + beforePixels := desktop.desktopResponse.Snapshot.ScreenshotPNGBase64 + response, err := DecodeProviderWireResponse([]byte(fmt.Sprintf(`{ + "protocol":%q,"schema_version":%d,"request_id":"set-value","operation":"resolve_and_act", + "accessibility":{"ok":false, + "error":"effect_unknown: the original editor did not confirm the requested value; text was not retried. Observe before sending more input.", + "failure":{"code":"effect_unknown","phase":"readback","subject":"element"}, + "action_evidence":{"route":"keyboard_synthesis","input_emitted":true,"geometry_sampled":false,"readback_sampled":true,"readback_verified":false}} + }`, ProviderProtocolName, ProviderProtocolVersion))) + if err != nil { + t.Fatalf("decode existing failure wire: %v", err) + } + backend := &unconfirmedSetValueBackend{reportBackend: desktop, + failure: *response.Accessibility, failAt: failAt, snapshotInFailure: scenario.snapshotInFailure} + service.Backend = backend + desktop.onAction = func(current *reportBackend) { + if len(current.actionRequests) != failAt { + current.checked = true + return + } + if scenario.focusChanged { + current.textStates = "[editable focusable focused]" + } + if scenario.pixelsChanged { + current.page = "Editing" + pixels := image.NewRGBA(image.Rect(0, 0, 40, 30)) + pixels.SetRGBA(10, 10, color.RGBA{R: 255, A: 255}) + var encoded bytes.Buffer + if err := png.Encode(&encoded, pixels); err != nil { + t.Fatal(err) + } + current.desktopResponse = testDesktopFrame(t, "after", 40, 30) + current.desktopResponse.Snapshot.ScreenshotPNGBase64 = base64.StdEncoding.EncodeToString(encoded.Bytes()) + backend.failure.Notes = []string{"[pixels] 1% of the window changed on screen."} + } + } + actions := []any{} + if failAt == 2 { + actions = append(actions, map[string]any{"type": "click", "ref": refs["Interface"]}) + } + actions = append(actions, + map[string]any{"type": "set_value", "ref": refs["Size"], "value": "16"}, + map[string]any{"type": "wait", "ref": refs["Size"], "value": "12", "timeout_ms": 1.0}, + map[string]any{"type": "keypress", "keys": []any{"ENTER"}}, + ) + result := service.callTool(contract.PublicToolAX, map[string]any{ + "mode": "act", "window_ref": window, "observe": observe, "actions": actions, + }) + if len(backend.actionRequests) != failAt { + t.Fatalf("failed input was retried or later input was sent: %#v", backend.actionRequests) + } + if request := backend.actionRequests[failAt-1]; request.Intent != AccessibilitySetValue || request.SetValue == nil || request.SetValue.Value == nil || *request.SetValue.Value != "16" { + t.Fatalf("failure did not exercise set_value: %#v", request) + } + full := result.StructuredContent + batch, _ := full["a11y_batch_result"].(map[string]any) + entries, _ := batch["actions"].([]map[string]any) + if len(entries) != failAt || entries[failAt-1]["call_error"] != true || + batch["failed_action_index"] != failAt-1 || batch["failed_action_type"] != "set_value" { + t.Fatalf("failure or batch stop was lost: %#v", batch) + } + for _, evidence := range []map[string]any{full, entries[failAt-1]} { + runtime, ok := evidence["runtime_action_evidence"].(RuntimeActionEvidence) + if !ok || runtime.Route != "keyboard_synthesis" || !runtime.InputEmitted || + !runtime.ReadbackSampled || runtime.ReadbackVerified == nil || *runtime.ReadbackVerified || + evidence["input_events_emitted"] != true || evidence["mutation_unknown"] != true { + t.Fatalf("unconfirmed runtime input evidence was lost: %#v", evidence) + } + } + prepared := PrepareAXEvidence(result) + public := prepared.StructuredContent + if !prepared.IsError || public["ok"] != false || public["code"] != contract.AXCodeEffectUnknown || + public["completed"] != failAt-1 || public["requested"] != len(actions) || public["input_emitted"] != true || + public["failure_phase"] != "readback" { + t.Fatalf("changed state overrode provider failure: %#v\n%s", public, ResultText(prepared)) + } + continuation, _ := public["continuation"].(map[string]any) + if continuation["state"] != "verify_only" || continuation["retry_same_request"] != false { + t.Fatalf("unsafe continuation after unconfirmed input: %#v", continuation) + } + for _, required := range []string{"state=verify_only", "retry_same_request=false", "text was not retried", "Observe before sending more input"} { + if !strings.Contains(ResultText(prepared), required) { + t.Errorf("model text lost %q: %s", required, ResultText(prepared)) + } + } + after := desktop.snapshot(reportMain()) + if desktop.value != "12" || (after.Elements[2].States != before.Elements[2].States) != scenario.focusChanged || + (desktop.desktopResponse.Snapshot.ScreenshotPNGBase64 != beforePixels) != scenario.pixelsChanged { + t.Fatal("fixture did not keep the original value while changing only the requested focus/pixels") + } + if observe == contract.ObserveFull { + if full["recovery_observation_returned"] != true || full["task_effect_verified"] != false || full["cause_unattributed"] != true { + t.Fatalf("fresh observation falsely confirmed the failed action: %#v", full) + } + if onlyImage(t, prepared).Data != desktop.desktopResponse.Snapshot.ScreenshotPNGBase64 { + t.Fatal("failure did not return the current pixels") + } + if scenario.pixelsChanged && !strings.Contains(ResultText(prepared), "Editing") { + t.Fatalf("failure did not return the changed editor: %s", ResultText(prepared)) + } + } else if len(desktop.observeRequests) != 1 || len(desktop.desktopRequests) != 0 { + t.Fatal("observe none unexpectedly read the desktop or ran the wait") + } + }) + } + } + } +} + +func TestInvalidKeyKeepsPublicArgumentErrorAndAcceptedSpelling(t *testing.T) { + response := ProviderAccessibilityResult{ + Error: "Unsupported key: KP_AD. Use KP_ADD.", + Failure: &ProviderFailure{ + Code: "invalid_argument", Phase: "precondition", FactsComplete: true, + }, + } + result := NewService().runtimeReportedFailure("", AXServiceRequest{}, response) + result = prependAXBatchSummary(result, "Stopped at action 1/1 (keypress).", map[string]any{ + "actions_requested": 1, "actions_completed_without_call_error": 0, + }) + public := axPublicEvidence(result.StructuredContent, result.IsError, result.Content) + if public["code"] != "invalid_argument" || public["input_emitted"] != false || public["completed"] != 0 { + t.Fatalf("invalid key classification lost: %#v", public) + } + if message, _ := public["message"].(string); !strings.Contains(message, "Use KP_ADD") { + t.Fatalf("accepted spelling lost behind batch summary: %#v", public) + } +} diff --git a/computer-use/internal/core/ax_snapshot.go b/computer-use/internal/core/ax_snapshot.go index 6557605..7f680b3 100644 --- a/computer-use/internal/core/ax_snapshot.go +++ b/computer-use/internal/core/ax_snapshot.go @@ -61,10 +61,12 @@ type AXCapabilities struct { } type AXProperties struct { - Value string `json:"value,omitempty"` - Description string `json:"description,omitempty"` - Placeholder string `json:"placeholder,omitempty"` - Omitted string `json:"omitted,omitempty"` + TextTruncated []string `json:"textTruncated,omitempty"` + TextLengths map[string]int `json:"textLengths,omitempty"` + Value string `json:"value,omitempty"` + Description string `json:"description,omitempty"` + Placeholder string `json:"placeholder,omitempty"` + Omitted string `json:"omitted,omitempty"` } type AXProviderIdentity struct { diff --git a/computer-use/internal/core/ax_snapshot_build.go b/computer-use/internal/core/ax_snapshot_build.go index 5c3670e..3528788 100644 --- a/computer-use/internal/core/ax_snapshot_build.go +++ b/computer-use/internal/core/ax_snapshot_build.go @@ -231,9 +231,11 @@ func newAXNodeFromRecord(snapshot *AppSnapshot, record *ElementRecord, binding a Name: normalizedTreeText(record.Name), States: typedAXStates(record), Properties: AXProperties{ - Value: normalizedTreeText(record.Value), - Description: normalizedTreeText(record.Description), - Placeholder: normalizedTreeText(record.Placeholder), + TextTruncated: append([]string(nil), record.TextTruncated...), + TextLengths: axTextLengths(record), + Value: normalizedTreeText(record.Value), + Description: normalizedTreeText(record.Description), + Placeholder: normalizedTreeText(record.Placeholder), }, Relations: record.Relations, NameProvenance: AXNameProvenance{ diff --git a/computer-use/internal/core/ax_snapshot_refresh.go b/computer-use/internal/core/ax_snapshot_refresh.go index beb9171..3cbb505 100644 --- a/computer-use/internal/core/ax_snapshot_refresh.go +++ b/computer-use/internal/core/ax_snapshot_refresh.go @@ -6,6 +6,8 @@ import ( "strconv" "strings" "time" + + "github.com/microsoft/Hybrid-Computer-Use/computer-use/internal/contract" ) func (s *Service) currentSnapshot(app string) *AppSnapshot { @@ -148,6 +150,14 @@ func (s *Service) prepareProviderSnapshot( response.Snapshot, request.ObservationRoot, ); err != nil { + var lifecycle *axRefLifecycleError + if errors.As(err, &lifecycle) { + result := withAXLifecycleFailure(TextResult(err.Error(), true), err) + result.StructuredContent[contract.AXPublicErrorPathKey] = "root" + result.StructuredContent[contract.AXPublicErrorHintKey] = contract.CompactJSON( + map[string]any{"mode": contract.AXModeObserve, "app": request.App}) + return nil, result.withEvidenceScope("error_only") + } return nil, TextResult(err.Error(), true). withEvidenceScope("error_only") } diff --git a/computer-use/internal/core/ax_snapshot_refresh_test.go b/computer-use/internal/core/ax_snapshot_refresh_test.go index d9ea1a5..0debe1c 100644 --- a/computer-use/internal/core/ax_snapshot_refresh_test.go +++ b/computer-use/internal/core/ax_snapshot_refresh_test.go @@ -6,6 +6,43 @@ import ( "time" ) +func TestProviderBoundRootFailurePreservesPublicError(t *testing.T) { + for _, failure := range []string{"missing", "ambiguous", "different window"} { + t.Run(failure, func(t *testing.T) { + service := NewServiceWithBackend(&recordingPlatformBackend{}) + original := anonymousRootFixture("panel", true) + if err := service.BuildAndRenderAXSnapshot(original, "working"); err != nil { + t.Fatal(err) + } + found := service.freezeFindRefs(original, []ElementRecord{original.Elements[1]}) + root, err := service.expandObservationRootShortRef("Editor", map[string]any{"ref": found.AXSnapshot.ByIndex[1].Ref}) + if err != nil { + t.Fatal(err) + } + snapshot := anonymousRootFixture("panel", true) + switch failure { + case "missing": + snapshot.Elements = append(snapshot.Elements[:1], snapshot.Elements[2:]...) + case "ambiguous": + snapshot.Elements = append(snapshot.Elements, snapshot.Elements[1]) + case "different window": + snapshot.WindowHandle = reportOtherHandle + } + _, result := service.prepareProviderSnapshot(AXServiceRequest{ + App: "Editor", ObservationRoot: root, + }, &ProviderAccessibilityResult{OK: true, Snapshot: snapshot}) + public := PrepareAXEvidence(result).StructuredContent + if !result.IsError || public["code"] != "stale_ref" || public["input_emitted"] != false || public["path"] != "root" { + t.Errorf("lost root failure identity: %#v", public) + } + hint := decodeJSONForTest(t, public["hint"].(string)) + if hint["mode"] != "observe" || hint["app"] != "Editor" || hint["root"] != nil { + t.Errorf("root recovery must observe the app without the invalid root: %#v", hint) + } + }) + } +} + func TestActionResultFlagsUnchangedState(t *testing.T) { base := func() *AppSnapshot { return &AppSnapshot{ diff --git a/computer-use/internal/core/ax_text_read.go b/computer-use/internal/core/ax_text_read.go new file mode 100644 index 0000000..540d90c --- /dev/null +++ b/computer-use/internal/core/ax_text_read.go @@ -0,0 +1,171 @@ +package core + +import ( + "fmt" + "runtime" + "unicode/utf8" + + "github.com/microsoft/Hybrid-Computer-Use/computer-use/internal/contract" +) + +func decodeAXTextRead(raw any) (*ProviderTextRead, *axArgError) { + object, ok := raw.(map[string]any) + if !ok { + return nil, axInvalid("read_text", "read_text must be an object", "").withFix(axSet("read_text", map[string]any{"field": "value"})) + } + if err := axCheckFields(object, []string{"field", "start", "limit"}, "read_text", "read_text"); err != nil { + return nil, err + } + field, err := axOptionalEnum(object, "field", "read_text", []string{"name", "value", "description"}) + if err != nil { + return nil, err + } + if field == "" { + return nil, axInvalid("read_text.field", "read_text.field is required", "").withFix(axSet("read_text.field", "value")) + } + read := &ProviderTextRead{Field: field, Limit: contract.AXMaxReadText} + for _, item := range []struct { + name string + minimum int + maximum int + target *int + }{{"start", 0, int(^uint32(0) >> 1), &read.Start}, {"limit", 1, contract.AXMaxReadText, &read.Limit}} { + if value, present := object[item.name]; present { + number, valid := axInteger(value) + if !valid || number < item.minimum || number > item.maximum { + path := "read_text." + item.name + return nil, axInvalid(path, fmt.Sprintf("%s must be an integer from %d to %d", path, item.minimum, item.maximum), "").withFix(axNumberEdit(path, value, float64(item.minimum), float64(item.maximum), true)) + } + *item.target = number + } + } + return read, nil +} + +func axTextReadError(message string) ToolCallResult { + result := TextResult(message, true) + result.StructuredContent = map[string]any{ + "code": contract.AXCodeRuntimeError, "message": message, + "input_events_emitted": false, "desktop_state_mutated": false, + "mutation_unknown": false, "evidence_scope": "error_only", + } + return result +} + +func (s *Service) axReadText(request *axRequest) ToolCallResult { + if runtime.GOOS != "linux" { + return axTextReadError("read_text is supported only by the Linux provider") + } + identity, err := s.resolveAXReadIdentity(request.app, request.windowRef, request.root) + if err != nil { + return axIdentityErrorResult(err) + } + entry, err := s.resolveShortAXRefOwned(request.root) + if err != nil { + return axIdentityErrorResult(err) + } + target, err := semanticTargetFromRefEntry(entry) + if err != nil { + return axStaleRefResult(request.root, err) + } + read := *request.readText + available := MaxInlineChars() - 3072 + if available < 1 { + return axTextReadError("read_text needs COMPUTER_USE_MAX_INLINE_CHARS above 3072 to deliver a text page and its continuation") + } + if read.Limit > available { + read.Limit = available + } + no := false + operation := AXServiceRequest{ + Operation: axServiceObserve, App: identity.appToken, + TargetWindowHandle: identity.windowHandle, SemanticTarget: target, + SourceRef: request.root, Action: axResolveTargetCommand{}, + TextRead: &read, CaptureSnapshot: &no, IncludeScreenshot: &no, + } + if err := s.bindRuntimeAppIdentity(&operation); err != nil { + return axIdentityErrorResult(err) + } + response, err := s.callProviderAction(providerActionRequestFromOperation(operation)) + if err != nil { + return axTextReadError(err.Error()) + } + if !response.OK { + return s.runtimeReportedFailure(identity.appToken, operation, *response) + } + page := response.TextRead + if page == nil || page.Field != read.Field || page.Start != read.Start || + page.End < page.Start || page.End > page.Total || + page.End-page.Start > read.Limit || !utf8.ValidString(page.Text) || + utf8.RuneCountInString(page.Text) != page.End-page.Start || + (page.End < page.Total && page.End-page.Start != read.Limit) { + return axTextReadError("the provider did not return a complete, valid text page") + } + metadata := map[string]any{ + "ref": request.root, "window_ref": identity.windowRef, + "field": page.Field, "start": page.Start, "end": page.End, "total": page.Total, + "truncated": page.End < page.Total, + } + if page.End < page.Total { + metadata["next_read"] = axTextReadCall(identity.windowRef, request.root, page.Field, page.End, request.readText.Limit) + } + result := TextResult(contract.CompactJSON(metadata)+"\n"+page.Text, false) + result.StructuredContent = map[string]any{ + "text_page": page, "text_read": metadata, + "input_events_emitted": false, "desktop_state_mutated": false, + "mutation_unknown": false, "evidence_scope": "independent_ui_read", + } + return result +} + +func axTextReadCall(window, ref, field string, start, limit int) map[string]any { + call := map[string]any{ + "mode": "observe", "root": ref, + "read_text": map[string]any{"field": field, "start": start, "limit": limit}, + } + if window != "" { + call["window_ref"] = window + } + return call +} + +func axTextLengths(record *ElementRecord) map[string]int { + lengths := make(map[string]int, 3) + for field, value := range map[string]string{"name": record.Name, "value": record.Value, "description": record.Description} { + length := utf8.RuneCountInString(value) + if full, known := record.TextLengths[field]; known { + length = full + } + if length > 0 { + lengths[field] = length + } + } + return lengths +} + +func axTextPreviewTruncated(node *AXNode, field, preview string) bool { + if contract.ContainsString(node.Properties.TextTruncated, field) { + return true + } + original := "" + if record := node.SourceRecord; record != nil { + switch field { + case "name": + original = record.Name + case "value": + original = record.Value + case "description": + original = record.Description + } + } else { + switch field { + case "name": + original = node.Name + case "value": + original = node.Properties.Value + case "description": + original = node.Properties.Description + } + } + return utf8.RuneCountInString(normalizedTreeText(original)) > utf8.RuneCountInString(preview) +} diff --git a/computer-use/internal/core/ax_text_read_test.go b/computer-use/internal/core/ax_text_read_test.go new file mode 100644 index 0000000..9ed62c0 --- /dev/null +++ b/computer-use/internal/core/ax_text_read_test.go @@ -0,0 +1,171 @@ +package core + +import ( + "errors" + "strings" + "testing" + "unicode/utf8" + + "github.com/microsoft/Hybrid-Computer-Use/computer-use/internal/contract" +) + +func textReadService(t *testing.T) (*Service, *recordingPlatformBackend, string) { + t.Helper() + snapshot := windowTransitionSnapshot("w2:0123456789abcdef0123456789abcdef", "Document", true) + backend := &recordingPlatformBackend{accessibilityResponse: &ProviderAccessibilityResult{OK: true, Snapshot: snapshot}} + service := NewServiceWithBackend(backend) + result := service.callTool("AX", map[string]any{"mode": "observe", "app": "Editor"}) + if result.IsError { + t.Fatal(ResultText(result)) + } + ref := service.currentSnapshot("Editor").AXSnapshot.ByIndex[3].Ref + if ref == "" { + t.Fatal("setup did not issue a ref") + } + backend.calls = nil + return service, backend, ref +} + +func TestTextReadPagesKeepUnicodeAndLiveIdentity(t *testing.T) { + service, backend, ref := textReadService(t) + body := strings.Repeat("中🙂e\u0301", 1100) + runes := []rune(body) + backend.accessibilityResponses = []ProviderAccessibilityResult{ + {OK: true, TextRead: &ProviderTextPage{Field: "value", Start: 0, End: 4000, Total: len(runes), Text: string(runes[:4000])}}, + {OK: true, TextRead: &ProviderTextPage{Field: "value", Start: 4000, End: len(runes), Total: len(runes), Text: string(runes[4000:])}}, + } + result := service.callTool("AX", axTextReadCall("", ref, "value", 0, 4000)) + if result.IsError { + t.Fatal(ResultText(result)) + } + page := result.StructuredContent["text_page"].(*ProviderTextPage) + metadata := result.StructuredContent["text_read"].(map[string]any) + next := metadata["next_read"].(map[string]any) + result = service.callTool("AX", next) + if result.IsError { + t.Fatal(ResultText(result)) + } + last := result.StructuredContent["text_page"].(*ProviderTextPage) + if page.Text+last.Text != body || result.StructuredContent["text_read"].(map[string]any)["next_read"] != nil { + t.Fatal("text paging lost characters or left a false continuation") + } + if len(backend.calls) != 2 || len(backend.actionRequests) != 2 { + t.Fatalf("text reads took unrelated snapshots: %v", backend.calls) + } + for _, request := range backend.actionRequests { + if request.Intent != AccessibilityResolveTarget || request.Target.ProviderObjectPath != "/dialog/3" || + request.Target.WindowHandle == "" || request.AppIdentity == nil || + request.Observation.CaptureSnapshot == nil || *request.Observation.CaptureSnapshot || + request.Observation.IncludeScreenshot == nil || *request.Observation.IncludeScreenshot { + t.Fatalf("read lost identity or requested side effects: %#v", request) + } + } +} + +func TestTextReadRecordsUseTheSameBoundedProviderRead(t *testing.T) { + service, backend, ref := textReadService(t) + backend.accessibilityResponse = &ProviderAccessibilityResult{OK: true, TextRead: &ProviderTextPage{Field: "value", Start: 0, End: 3, Total: 3, Text: "中🙂e"}} + record := service.CallRecord("AX", axTextReadCall("", ref, "value", 0, 3), CallOptions{}) + if !record.OK || record.Evidence["text_page"] == nil || len(backend.calls) != 1 || backend.calls[0] != "resolve_and_act" { + t.Fatalf("records path did not read the same text page: %#v %v", record, backend.calls) + } +} + +func TestTextReadRejectsWrongWindowAndStaleRefBeforeProvider(t *testing.T) { + service, backend, ref := textReadService(t) + for _, call := range []map[string]any{ + axTextReadCall("w999", ref, "value", 0, 1), + axTextReadCall("", "r999", "value", 0, 1), + } { + result := service.callTool("AX", call) + if !result.IsError || len(backend.calls) != 0 { + t.Fatalf("invalid owner reached provider: %#v %v", result, backend.calls) + } + } +} + +func TestTextReadRejectsAbsentShortOrMalformedProviderPages(t *testing.T) { + for _, page := range []*ProviderTextPage{ + nil, + {Field: "value", Start: 0, End: 0, Total: 5}, + {Field: "value", Start: 0, End: 3, Total: 3, Text: "ab"}, + {Field: "name", Start: 0, End: 3, Total: 3, Text: "abc"}, + {Field: "value", Start: 1, End: 3, Total: 3, Text: "ab"}, + } { + service, backend, ref := textReadService(t) + backend.accessibilityResponse = &ProviderAccessibilityResult{OK: true, TextRead: page} + result := service.callTool("AX", axTextReadCall("", ref, "value", 0, 3)) + if !result.IsError || result.StructuredContent["code"] != contract.AXCodeRuntimeError { + t.Fatalf("invalid page reported success: %#v", result) + } + } +} + +func TestTextReadPropagatesProviderFailures(t *testing.T) { + service, backend, ref := textReadService(t) + backend.err = errors.New("Text interface read failed") + result := service.callTool("AX", axTextReadCall("", ref, "value", 0, 2)) + if !result.IsError || !strings.Contains(ResultText(result), "Text interface read failed") { + t.Fatalf("read error hidden: %#v", result) + } + backend.err = nil + backend.accessibilityResponse = &ProviderAccessibilityResult{OK: false, Error: "stale target", Failure: &ProviderFailure{Code: "stale_target", Phase: "precondition"}} + result = service.callTool("AX", axTextReadCall("", ref, "value", 0, 2)) + if !result.IsError || axPublicErrorCode(result.StructuredContent) != contract.AXCodeStaleRef { + t.Fatalf("stale live target hidden: %#v", result) + } +} + +func TestTextReadBudgetDoesNotCutDeliveredPage(t *testing.T) { + t.Setenv("COMPUTER_USE_MAX_INLINE_CHARS", "4096") + service, backend, ref := textReadService(t) + body := "[TRUNCATED: this is application text]\n" + strings.Repeat("🙂", 1100) + runes := []rune(body) + backend.accessibilityResponse = &ProviderAccessibilityResult{OK: true, TextRead: &ProviderTextPage{ + Field: "value", Start: 0, End: 1024, Total: len(runes), Text: string(runes[:1024]), + }} + result := service.callTool("AX", axTextReadCall("", ref, "value", 0, 4000)) + if result.IsError || backend.actionRequests[0].Observation.TextRead.Limit != 1024 || + result.StructuredContent["truncation_reason"] == "response_budget" || + !strings.Contains(ResultText(result), string(runes[:1024])) || utf8.RuneCountInString(ResultText(result)) > 4096 { + t.Fatalf("budget damaged page or continuation: %#v", result) + } + for _, structured := range []bool{false, true} { + final := FinalizeForModel("AX", result, structured) + if !strings.Contains(ResultText(final), string(runes[:1024])) || utf8.RuneCountInString(ResultText(final)) > 4096 { + t.Fatalf("model finalization damaged text: %#v", final) + } + if structured && final.StructuredContent["text_page"] == nil { + t.Fatal("structured consumers lost the text page") + } + } +} + +func TestProviderTruncationIsMarkedEvenWhenEllipsisFillsTheGap(t *testing.T) { + for _, total := range []int{501, 502, 503} { + record := &ElementRecord{Name: strings.Repeat("x", 500) + "...", TextLengths: map[string]int{"name": total}, TextTruncated: []string{"name"}} + node := &AXNode{Name: record.Name, Ref: "r1", SourceRecord: record, Properties: AXProperties{TextLengths: axTextLengths(record), TextTruncated: record.TextTruncated}} + found := false + for _, property := range renderAXProperties(node, AXRenderOptions{}) { + if property[0] == "/text-truncated" { + found = true + if !strings.Contains(property[1], "read_text") { + t.Fatal("missing recovery route") + } + } + } + if !found { + t.Fatalf("total=%d was silently clipped", total) + } + } +} + +func TestWhitespaceNormalizationIsNotReportedAsTruncation(t *testing.T) { + record := &ElementRecord{Value: "hello\n\nworld", TextLengths: map[string]int{"value": 12}} + node := &AXNode{Role: "paragraph", SourceRecord: record, Properties: AXProperties{Value: "hello world", TextLengths: axTextLengths(record)}} + for _, property := range renderAXProperties(node, AXRenderOptions{}) { + if property[0] == "/text-truncated" { + t.Fatal("whitespace normalization was reported as truncation") + } + } +} diff --git a/computer-use/internal/core/ax_wait_batch.go b/computer-use/internal/core/ax_wait_batch.go index c3cc59b..2398ef4 100644 --- a/computer-use/internal/core/ax_wait_batch.go +++ b/computer-use/internal/core/ax_wait_batch.go @@ -32,8 +32,9 @@ func (s *Service) preflightAXWaitRef(context *AXBatchContext, ref string) error return err } if context != nil && entry.WindowHandle != context.WindowHandle { - return fmt.Errorf("%s belongs to window %s, not the batch window %s; wait for it in its own act call", - ref, entry.LogicalWindowRef, context.WindowRef) + return axInvalid("", fmt.Sprintf("%s belongs to window %s, not the current batch window %s; activate that window before using its refs", + ref, entry.LogicalWindowRef, context.WindowRef), + "Use refs from the current window, or activate the other window before acting in it.") } return nil } @@ -88,6 +89,10 @@ func (s *Service) rebindAfterWindowWait( Role: entry.ProviderWindowRole, Name: entry.ProviderWindowName, } + if context.report != nil { + context.report.appToken = context.AppToken + context.report.windowRef = context.WindowRef + } // One read of the new window makes it the current state of its app, so // the next step's window check and its diff start from that window. noScreenshot := false diff --git a/computer-use/internal/core/provider_request_builder.go b/computer-use/internal/core/provider_request_builder.go index 077d6df..50aa88e 100644 --- a/computer-use/internal/core/provider_request_builder.go +++ b/computer-use/internal/core/provider_request_builder.go @@ -92,6 +92,7 @@ func providerObservationOptionsFromOperation( // the read (attachDesktopFrame), so the provider never captures one. noImage := false return ProviderObservationOptions{ + TextRead: operation.TextRead, TextLimit: providerTextLimit(operation.TextLimit), MaxTreeNodes: maxTreeNodes, MaxTreeDepth: operation.MaxTreeDepth, diff --git a/computer-use/internal/core/provider_types.go b/computer-use/internal/core/provider_types.go index b7787f1..e86ca7e 100644 --- a/computer-use/internal/core/provider_types.go +++ b/computer-use/internal/core/provider_types.go @@ -85,7 +85,22 @@ type ProviderSemanticTarget struct { SourceIndex *int `json:"-"` } +type ProviderTextRead struct { + Field string `json:"field"` + Start int `json:"start"` + Limit int `json:"limit"` +} + +type ProviderTextPage struct { + Field string `json:"field"` + Start int `json:"start"` + End int `json:"end"` + Total int `json:"total"` + Text string `json:"text"` +} + type ProviderObservationOptions struct { + TextRead *ProviderTextRead `json:"text_read,omitempty"` TextLimit *ProviderTextLimit `json:"text_limit,omitempty"` MaxTreeNodes int `json:"max_tree_nodes,omitempty"` MaxTreeDepth int `json:"max_tree_depth,omitempty"` @@ -153,17 +168,19 @@ func (limit *ProviderTextLimit) UnmarshalJSON(data []byte) error { } type ProviderClickInput struct { - X *float64 `json:"x,omitempty"` - Y *float64 `json:"y,omitempty"` - ClickCount int `json:"click_count,omitempty"` - MouseButton string `json:"mouse_button,omitempty"` - ClickMethod string `json:"click_method,omitempty"` + AllowGlobalInput *bool `json:"allow_global_input,omitempty"` + X *float64 `json:"x,omitempty"` + Y *float64 `json:"y,omitempty"` + ClickCount int `json:"click_count,omitempty"` + MouseButton string `json:"mouse_button,omitempty"` + ClickMethod string `json:"click_method,omitempty"` } type ProviderScrollInput struct { - Direction string `json:"direction,omitempty"` - Pages float64 `json:"pages,omitempty"` - WindowCentre bool `json:"window_centre,omitempty"` + AllowGlobalInput *bool `json:"allow_global_input,omitempty"` + Direction string `json:"direction,omitempty"` + Pages float64 `json:"pages,omitempty"` + WindowCentre bool `json:"window_centre,omitempty"` } type ProviderTypeInput struct { @@ -295,6 +312,17 @@ func (request *ProviderActionRequest) UnmarshalJSON(data []byte) error { if len(wire.Input) == 0 { return errors.New("provider action input is required") } + if wire.Intent == AccessibilityClick || wire.Intent == AccessibilityScroll { + var policy struct { + AllowGlobalInput json.RawMessage `json:"allow_global_input"` + } + if err := json.Unmarshal(wire.Input, &policy); err != nil { + return err + } + if string(policy.AllowGlobalInput) == "null" { + return errors.New("provider allow_global_input must be a boolean") + } + } return DecodeProviderWireJSON(wire.Input, target) } switch wire.Intent { @@ -435,6 +463,7 @@ type ProviderInventoryIssue struct { } type ProviderAccessibilityResult struct { + TextRead *ProviderTextPage `json:"text_read,omitempty"` OK bool `json:"ok"` Text string `json:"text,omitempty"` Error string `json:"error,omitempty"` diff --git a/computer-use/internal/core/provider_wire.go b/computer-use/internal/core/provider_wire.go index 226ebf7..58e3d9a 100644 --- a/computer-use/internal/core/provider_wire.go +++ b/computer-use/internal/core/provider_wire.go @@ -413,6 +413,9 @@ func ValidateProviderObserveAppsRequest(request ProviderObserveAppsRequest) erro } func ValidateProviderObserveRequest(request ProviderObserveRequest) error { + if request.Observation.TextRead != nil { + return errors.New("text_read requires resolve_target") + } if err := validateProviderProtocolVersion(request.ProtocolVersion); err != nil { return err } @@ -425,6 +428,12 @@ func ValidateProviderObserveRequest(request ProviderObserveRequest) error { func validateProviderObservationOptions( observation ProviderObservationOptions, ) error { + if read := observation.TextRead; read != nil { + if !contract.ContainsString([]string{"name", "value", "description"}, read.Field) || + read.Start < 0 || read.Start > 2147483647 || read.Limit < 1 || read.Limit > contract.AXMaxReadText { + return errors.New("invalid provider text_read field, start, or limit") + } + } if observation.TextLimit != nil && !observation.TextLimit.Max && observation.TextLimit.Value <= 0 { @@ -471,6 +480,9 @@ func ValidateProviderActionRequest(request ProviderActionRequest) error { if !validAccessibilityOperation(request.Intent) { return fmt.Errorf("unsupported provider accessibility intent %q", request.Intent) } + if request.Observation.TextRead != nil && request.Intent != AccessibilityResolveTarget { + return errors.New("text_read requires resolve_target") + } if strings.TrimSpace(request.App) == "" { return errors.New("provider accessibility app is required") } @@ -735,6 +747,9 @@ func ValidateProviderWindowListRequest(request ProviderWindowListRequest) error func ValidateProviderWindowActionRequest( request ProviderWindowActionRequest, ) error { + if request.Observation.TextRead != nil { + return errors.New("text_read requires resolve_target") + } if err := validateProviderProtocolVersion(request.ProtocolVersion); err != nil { return err } diff --git a/computer-use/internal/core/provider_wire_test.go b/computer-use/internal/core/provider_wire_test.go index 655d5e0..e26c3d2 100644 --- a/computer-use/internal/core/provider_wire_test.go +++ b/computer-use/internal/core/provider_wire_test.go @@ -13,6 +13,31 @@ import ( var providerFixtureFS = os.DirFS("../..").(fs.ReadFileFS) +func TestLinuxTextReadWireFixtures(t *testing.T) { + requestData := readProviderFixture(t, "text-read.request.json") + request, err := DecodeProviderWireRequest(requestData) + if err != nil { + t.Fatal(err) + } + assertProviderFixtureRoundTrip(t, requestData, request) + responseData := readProviderFixture(t, "text-read.response.json") + response, err := DecodeProviderWireResponse(responseData) + if err != nil { + t.Fatal(err) + } + assertProviderFixtureRoundTrip(t, responseData, response) + for _, field := range []string{"value", "name", "description"} { + request.Action.Observation.TextRead.Field = field + if err := request.Validate(); err != nil { + t.Fatal(err) + } + } + request.Action.Observation.TextRead.Limit = contract.AXMaxReadText + 1 + if err := request.Validate(); err == nil { + t.Fatal("oversize text read accepted") + } +} + type providerFixtureManifest struct { SchemaVersion int `json:"schema_version"` Protocol string `json:"protocol"` @@ -45,6 +70,75 @@ func readProviderFixture(t *testing.T, name string) []byte { return data } +func TestProviderLinuxInputFixtureCorpus(t *testing.T) { + var manifest struct { + SchemaVersion int `json:"schema_version"` + Protocol string `json:"protocol"` + Platforms map[string]bool `json:"allow_global_input_platforms"` + Fixtures []providerFixtureEntry `json:"fixtures"` + } + if err := strictFixtureDecode(readProviderFixture(t, "linux-input-manifest.json"), &manifest); err != nil { + t.Fatal(err) + } + if manifest.Protocol != ProviderProtocolName || manifest.SchemaVersion != ProviderProtocolVersion || + !reflect.DeepEqual(manifest.Platforms, map[string]bool{"linux": true, "darwin": false, "windows": false}) { + t.Fatalf("invalid Linux input fixture contract: %#v", manifest) + } + seen := map[AccessibilityOperation]bool{} + for _, fixture := range manifest.Fixtures { + t.Run(fixture.Name, func(t *testing.T) { + data := readProviderFixture(t, fixture.File) + request, err := DecodeProviderWireRequest(data) + if err != nil { + t.Fatal(err) + } + if fixture.Direction != "request" || request.Operation != ProviderWireResolveAndAct || request.Action == nil { + t.Fatal("Linux input fixture must exercise an accessibility action") + } + assertProviderFixtureRoundTrip(t, data, request) + var policy *bool + switch request.Action.Intent { + case AccessibilityClick: + policy = request.Action.Click.AllowGlobalInput + case AccessibilityScroll: + policy = request.Action.Scroll.AllowGlobalInput + if request.Action.Scroll.Pages != 1.5 { + t.Fatal("scroll fixture must preserve fractional pages") + } + default: + t.Fatalf("unexpected fixture intent %s", request.Action.Intent) + } + if policy == nil || *policy { + t.Fatal("fixture must retain explicit false") + } + seen[request.Action.Intent] = true + for platform, supported := range manifest.Platforms { + for _, allowed := range []bool{false, true} { + *policy = allowed + backend := &recordingPlatformBackend{accessibilityResponse: &ProviderAccessibilityResult{OK: true}} + service := NewServiceWithBackend(&globalInputPolicyBackend{backend, platform}) + result, err := service.callProviderAction(*request.Action) + if err != nil { + t.Fatal(err) + } + if supported { + if !result.OK || len(backend.actionRequests) != 1 { + t.Fatalf("Linux fixture was not dispatched: %#v", result) + } + } else if result.OK || len(backend.actionRequests) != 0 || result.Failure == nil || + result.Failure.Code != "invalid_argument" || result.Failure.Phase != "precondition" || + result.Mutation == nil || result.Mutation.InputEventsEmitted { + t.Fatalf("%s silently accepted policy %t: %#v", platform, allowed, result) + } + } + } + }) + } + if !reflect.DeepEqual(seen, map[AccessibilityOperation]bool{AccessibilityClick: true, AccessibilityScroll: true}) { + t.Fatalf("missing Linux input fixtures: %v", seen) + } +} + func TestProviderNativeFixtureCorpus(t *testing.T) { var manifest providerNativeFixtureManifest if err := strictFixtureDecode(readProviderFixture(t, "native-manifest.json"), &manifest); err != nil { diff --git a/computer-use/internal/core/records.go b/computer-use/internal/core/records.go index 6fe5695..f53f25b 100644 --- a/computer-use/internal/core/records.go +++ b/computer-use/internal/core/records.go @@ -320,8 +320,8 @@ func (s *Service) observeRecords(args map[string]any, since string) (Record, boo if bad { return errorRecord(contract.PublicToolAX, rejected), true } - if request.observe == contract.ObserveScreenshot || request.observe == contract.ObserveFull { - // Screenshots go through the shared path and come back as files. + if request.readText != nil || request.observe == contract.ObserveScreenshot || request.observe == contract.ObserveFull { + // Text pages and screenshots use the shared read path. return Record{}, false } readContext, identityErr := s.resolveAXReadIdentity(request.app, request.windowRef, "") diff --git a/computer-use/internal/core/runtime_backend.go b/computer-use/internal/core/runtime_backend.go index 28354f3..f97cd86 100644 --- a/computer-use/internal/core/runtime_backend.go +++ b/computer-use/internal/core/runtime_backend.go @@ -98,6 +98,17 @@ func (s *Service) callProviderAction( request ProviderActionRequest, ) (*ProviderAccessibilityResult, error) { defer s.countProviderTime(time.Now()) + if s.platformBackend().HostInfo().Platform != "linux" && + ((request.Click != nil && request.Click.AllowGlobalInput != nil) || + (request.Scroll != nil && request.Scroll.AllowGlobalInput != nil)) { + result := &ProviderAccessibilityResult{ + Error: "allow_global_input is supported only for Linux AX click and scroll", + Failure: &ProviderFailure{Code: "invalid_argument", Phase: "precondition", FactsComplete: true}, + Mutation: &ProviderMutationFacts{}, + } + result.PrepareForService() + return result, nil + } result, err := s.platformBackend().ResolveAndAct(request) result.PrepareForService() return &result, err diff --git a/computer-use/internal/core/runtime_backend_test.go b/computer-use/internal/core/runtime_backend_test.go index 84e7200..e4d2826 100644 --- a/computer-use/internal/core/runtime_backend_test.go +++ b/computer-use/internal/core/runtime_backend_test.go @@ -8,6 +8,28 @@ import ( "testing" ) +func TestProviderGlobalInputPolicyRejectsNonLinuxBeforeDispatch(t *testing.T) { + for _, platform := range []string{"darwin", "windows", "unknown"} { + for _, policy := range []bool{false, true} { + for _, intent := range []AccessibilityOperation{AccessibilityClick, AccessibilityScroll} { + backend := &recordingPlatformBackend{} + service := NewServiceWithBackend(&globalInputPolicyBackend{backend, platform}) + request := ProviderActionRequest{Intent: intent} + if intent == AccessibilityClick { + request.Click = &ProviderClickInput{AllowGlobalInput: &policy} + } else { + request.Scroll = &ProviderScrollInput{AllowGlobalInput: &policy} + } + result, err := service.callProviderAction(request) + if err != nil || result.OK || result.Failure == nil || result.Failure.Code != "invalid_argument" || + result.Failure.Phase != "precondition" || result.Mutation.InputEventsEmitted || len(backend.actionRequests) != 0 { + t.Fatalf("%s dispatched unsupported %s policy: %#v, %v", platform, intent, result, err) + } + } + } + } +} + func TestRuntimeCallsRouteByPlatformDomain(t *testing.T) { backend := &recordingPlatformBackend{ accessibilityResponse: &ProviderAccessibilityResult{OK: true}, diff --git a/computer-use/internal/core/runtime_protocol.go b/computer-use/internal/core/runtime_protocol.go index 7801102..4339d8c 100644 --- a/computer-use/internal/core/runtime_protocol.go +++ b/computer-use/internal/core/runtime_protocol.go @@ -17,6 +17,7 @@ type AXServiceRequest struct { WindowAction *WindowOperation `json:"-"` WindowBounds *frame `json:"windowBounds,omitempty"` TextLimit any `json:"text_limit,omitempty"` + TextRead *ProviderTextRead `json:"-"` MaxTreeNodes int `json:"max_tree_nodes,omitempty"` MaxTreeDepth int `json:"max_tree_depth,omitempty"` Prune *bool `json:"prune,omitempty"` @@ -66,11 +67,12 @@ func (axResolveTargetCommand) applyProviderAction(*ProviderActionRequest) {} func (axResolveTargetCommand) appendTraceFields(map[string]any) {} type AXClickCommand struct { - X *float64 - Y *float64 - ClickCount int - MouseButton string - ClickMethod string + AllowGlobalInput *bool + X *float64 + Y *float64 + ClickCount int + MouseButton string + ClickMethod string } func (AXClickCommand) Intent() AccessibilityOperation { @@ -81,15 +83,19 @@ func (command AXClickCommand) applyProviderAction( request *ProviderActionRequest, ) { request.Click = &ProviderClickInput{ - X: command.X, - Y: command.Y, - ClickCount: command.ClickCount, - MouseButton: command.MouseButton, - ClickMethod: command.ClickMethod, + AllowGlobalInput: command.AllowGlobalInput, + X: command.X, + Y: command.Y, + ClickCount: command.ClickCount, + MouseButton: command.MouseButton, + ClickMethod: command.ClickMethod, } } func (command AXClickCommand) appendTraceFields(fields map[string]any) { + if command.AllowGlobalInput != nil { + fields["allow_global_input"] = *command.AllowGlobalInput + } fields["x"] = command.X fields["y"] = command.Y fields["click_count"] = command.ClickCount @@ -98,9 +104,10 @@ func (command AXClickCommand) appendTraceFields(fields map[string]any) { } type axScrollCommand struct { - Direction string - Pages float64 - WindowCentre bool + AllowGlobalInput *bool + Direction string + Pages float64 + WindowCentre bool } func (axScrollCommand) Intent() AccessibilityOperation { @@ -111,13 +118,17 @@ func (command axScrollCommand) applyProviderAction( request *ProviderActionRequest, ) { request.Scroll = &ProviderScrollInput{ - Direction: command.Direction, - Pages: command.Pages, - WindowCentre: command.WindowCentre, + AllowGlobalInput: command.AllowGlobalInput, + Direction: command.Direction, + Pages: command.Pages, + WindowCentre: command.WindowCentre, } } func (command axScrollCommand) appendTraceFields(fields map[string]any) { + if command.AllowGlobalInput != nil { + fields["allow_global_input"] = *command.AllowGlobalInput + } fields["direction"] = command.Direction fields["pages"] = command.Pages fields["window_centre"] = command.WindowCentre diff --git a/computer-use/internal/core/runtime_protocol_test.go b/computer-use/internal/core/runtime_protocol_test.go new file mode 100644 index 0000000..6239ed2 --- /dev/null +++ b/computer-use/internal/core/runtime_protocol_test.go @@ -0,0 +1,69 @@ +package core + +import ( + "encoding/json" + "reflect" + "strings" + "testing" +) + +func TestAXGlobalInputPolicyWireRoundTrip(t *testing.T) { + for _, intent := range []AccessibilityOperation{AccessibilityClick, AccessibilityScroll} { + for _, value := range []*bool{nil, boolPointer(false), boolPointer(true)} { + var command axActionCommand = AXClickCommand{AllowGlobalInput: value} + if intent == AccessibilityScroll { + command = axScrollCommand{AllowGlobalInput: value, Direction: "down", Pages: 1} + } + request := ProviderActionRequest{ + ProtocolVersion: ProviderProtocolVersion, Intent: intent, App: "Editor", + Target: &ProviderSemanticTarget{Selector: `button "Save"`, HasName: true}, + } + command.applyProviderAction(&request) + wire := NewProviderWireRequest("policy", ProviderWireResolveAndAct) + wire.Action = &request + data, err := json.Marshal(wire) + if err != nil { + t.Fatal(err) + } + var raw struct { + Action struct { + Input map[string]any `json:"input"` + } `json:"action"` + } + if err := json.Unmarshal(data, &raw); err != nil { + t.Fatal(err) + } + serialized, present := raw.Action.Input["allow_global_input"] + if (value != nil) != present || (value != nil && serialized != *value) { + t.Fatalf("wire lost %s policy %v: %s", intent, value, data) + } + decoded, err := DecodeProviderWireRequest(data) + if err != nil { + t.Fatal(err) + } + var actual *bool + if intent == AccessibilityClick { + actual = decoded.Action.Click.AllowGlobalInput + } else { + actual = decoded.Action.Scroll.AllowGlobalInput + } + if !reflect.DeepEqual(actual, value) { + t.Fatalf("wire round trip lost %s policy: got %v, want %v", intent, actual, value) + } + trace := map[string]any{} + command.appendTraceFields(trace) + traced, present := trace["allow_global_input"] + if (value != nil) != present || (value != nil && traced != *value) { + t.Fatalf("trace lost %s policy: %v", intent, trace) + } + if value != nil && !*value { + for _, invalid := range []string{"null", " null ", "0", `"false"`, "[]", "{}"} { + invalidData := strings.Replace(string(data), `"allow_global_input":false`, `"allow_global_input":`+invalid, 1) + if _, err := DecodeProviderWireRequest([]byte(invalidData)); err == nil { + t.Fatalf("wire accepted invalid %s policy %s", intent, invalid) + } + } + } + } + } +} diff --git a/computer-use/internal/core/schema_consistency_test.go b/computer-use/internal/core/schema_consistency_test.go index 33c11ac..9c9f016 100644 --- a/computer-use/internal/core/schema_consistency_test.go +++ b/computer-use/internal/core/schema_consistency_test.go @@ -104,6 +104,10 @@ type probeReached struct{} type probeBackend struct{ *recordingPlatformBackend } +func (probeBackend) HostInfo() HostBackendInfo { + return HostBackendInfo{Platform: "linux"} +} + func (probeBackend) ObserveApps(ProviderObserveAppsRequest) (ProviderAccessibilityResult, error) { panic(probeReached{}) } diff --git a/computer-use/internal/core/tool_contract_test.go b/computer-use/internal/core/tool_contract_test.go index 435d10f..832b3bf 100644 --- a/computer-use/internal/core/tool_contract_test.go +++ b/computer-use/internal/core/tool_contract_test.go @@ -64,8 +64,14 @@ func TestMCPInstructionsRouteDesktopInputThroughTheTools(t *testing.T) { } } ax := findToolDefinition(t, "AX") - if !strings.Contains(ax.Description, "keypress/type without ref go to the focused window") { - t.Fatal("AX description must say where ref-less keys go") + for _, fragment := range []string{ + "keypress/type without ref use the batch target", + "initially the focused window unless refs or window_ref select it", + "activate with wN or title switches the batch target for later refs and ref-less steps", + } { + if !strings.Contains(ax.Description, fragment) { + t.Fatalf("AX description must explain ref-less input routing: missing %q", fragment) + } } } diff --git a/computer-use/internal/core/tool_result.go b/computer-use/internal/core/tool_result.go index a28a18f..cb13b81 100644 --- a/computer-use/internal/core/tool_result.go +++ b/computer-use/internal/core/tool_result.go @@ -3,6 +3,7 @@ package core import ( "encoding/base64" "fmt" + "maps" "os" "path/filepath" "strconv" @@ -182,6 +183,11 @@ func init() { // WithAXTextEvidence puts the AX evidence line first: one JSON object that // starts with ok, then code, path, message, hint and counts. func WithAXTextEvidence(result ToolCallResult) ToolCallResult { + if result.StructuredContent["text_page"] != nil { + result.StructuredContent = maps.Clone(result.StructuredContent) + delete(result.StructuredContent, "text_page") + delete(result.StructuredContent, "text_read") + } return withEvidenceLine(result, axPublicFields) } diff --git a/computer-use/internal/provider/linux_runtime_bundle.go b/computer-use/internal/provider/linux_runtime_bundle.go index b0e55ea..52ebead 100644 --- a/computer-use/internal/provider/linux_runtime_bundle.go +++ b/computer-use/internal/provider/linux_runtime_bundle.go @@ -13,6 +13,7 @@ var linuxProviderPaths = []string{ "native/linux/provider/x11_windows.py", "native/linux/provider/desktop_layout.py", "native/linux/provider/x11_window_inventory.py", + "native/linux/provider/x11_client_identity.py", "native/linux/provider/window_scale.py", "native/linux/provider/atspi_inventory.py", "native/linux/provider/ax_node_properties.py", @@ -42,6 +43,7 @@ var linuxProviderPaths = []string{ "native/linux/provider/input_method.py", "native/linux/provider/ax_actions.py", "native/linux/provider/ax_select.py", + "native/linux/provider/ax_semantic_scroll.py", "native/linux/provider/ax_dropdown.py", "native/linux/provider/ax_dispatch.py", "native/linux/provider/ax_input_delivery.py", diff --git a/computer-use/internal/provider/linux_runtime_bundle_test.go b/computer-use/internal/provider/linux_runtime_bundle_test.go index 87080f6..9b434da 100644 --- a/computer-use/internal/provider/linux_runtime_bundle_test.go +++ b/computer-use/internal/provider/linux_runtime_bundle_test.go @@ -28,6 +28,7 @@ func TestLinuxRuntimeBundleIsComplete(t *testing.T) { "native/linux/provider/ax_node_properties.py", "native/linux/provider/ax_records.py", "native/linux/provider/ax_select.py", + "native/linux/provider/ax_semantic_scroll.py", "native/linux/provider/ax_snapshot.py", "native/linux/provider/ax_snapshot_diagnostics.py", "native/linux/provider/ax_target_notes.py", @@ -64,6 +65,7 @@ func TestLinuxRuntimeBundleIsComplete(t *testing.T) { "native/linux/provider/window_scale.py", "native/linux/provider/wire.py", "native/linux/provider/wire_request.py", + "native/linux/provider/x11_client_identity.py", "native/linux/provider/x11_window_inventory.py", "native/linux/provider/x11_windows.py", "runtime.py", diff --git a/computer-use/internal/transport/contract_golden_test.go b/computer-use/internal/transport/contract_golden_test.go index 17855de..62f7c3b 100644 --- a/computer-use/internal/transport/contract_golden_test.go +++ b/computer-use/internal/transport/contract_golden_test.go @@ -123,6 +123,16 @@ var contractCalls = []struct { {"GUI", map[string]any{"actions": []any{map[string]any{"type": "screenshot"}}}}, {"Nope", map[string]any{}}, {" ax ", map[string]any{"mode": "observe", "observation_target": "apps"}}, + {"AX", map[string]any{"mode": "activate", "actions": []any{ + map[string]any{"type": "activate", "title": "Report"}, + }}}, + {"AX", map[string]any{"mode": "act", "actions": []any{ + map[string]any{"type": "click", "ref": "r1"}, + map[string]any{"type": "settle", "timeout_ms": 1200}, + }}}, + {"AX", map[string]any{"mode": "act", "actions": []any{ + map[string]any{"type": "double_click", "ref": "r1"}, + }}}, } type contractExchange struct { @@ -229,6 +239,22 @@ func capturePublicContract(t *testing.T) publicContract { }) } + snapshot := transportOrderedInputTestSnapshot() + snapshot.WindowHandle = transportTestHandleReport + snapshot.Elements = snapshot.Elements[:1] + snapshot.TreeLines = snapshot.TreeLines[:1] + contentService := core.NewServiceWithBackend(&transportRecordingPlatformBackend{ + accessibilityResponse: &core.ProviderAccessibilityResult{OK: true, Snapshot: snapshot}, + }) + arguments := map[string]any{"mode": "observe", "app": "Editor", "observe": "tree"} + contract.Calls = append(contract.Calls, contractExchange{ + Request: map[string]any{"tool": "AX", "arguments": arguments}, + Response: roundTripJSON(t, handleMCPRequest(map[string]any{ + "jsonrpc": "2.0", "id": float64(100), "method": "tools/call", + "params": map[string]any{"name": "AX", "arguments": arguments}, + }, contentService)), + }) + for _, args := range [][]string{ {"--help"}, {"help", "mcp"}, diff --git a/computer-use/native/linux/provider/ax_actions.py b/computer-use/native/linux/provider/ax_actions.py index e3fd5c3..9a72951 100644 --- a/computer-use/native/linux/provider/ax_actions.py +++ b/computer-use/native/linux/provider/ax_actions.py @@ -19,6 +19,7 @@ from .input_primitives import ( emit_keyboard_event, emit_mouse_event, + keypad_navigation_numlock_mask, require_window_focus, ) @@ -120,6 +121,21 @@ def normalize_key_expression(key): # Gdk.keyval_from_name returns VoidSymbol, not 0, for an unknown name. GDK_VOID_SYMBOL = 0xFFFFFF +GDK_KEY_NAMES = { + name[4:].lower(): getattr(Gdk, name) + for name in dir(Gdk) if name.startswith("KEY_") +} + + +def invalid_key(message): + return OperationFailure( + message + ". Use CTRL, SHIFT, ALT or SUPER before the main key; " + "use ENTER, TAB, ESC, PAGEUP/PAGE_UP, PAGEDOWN/PAGE_DOWN, " + "KP_ADD, KP_SUBTRACT, KP_ENTER or KP_0 through KP_9. " + "Use PLUS for the '+' key. Linux also accepts mapped X11 key names.", + failure_code="invalid_argument", failure_phase="precondition", + input_events_emitted=False, + ) def keysym(name): @@ -130,18 +146,12 @@ def keysym(name): case ("INSERT", "caps_lock"), because callers write keys the way they appear on the keyboard. """ - candidates = [name] - if len(name) > 1: - candidates.append("_".join(part.capitalize() for part in name.split("_"))) - # Punctuation keysyms are lowercase names ("comma", "plus"). - candidates.append(name.lower()) - for candidate in candidates: - sym = Gdk.keyval_from_name(candidate) - if sym and sym != GDK_VOID_SYMBOL: - return sym + sym = Gdk.keyval_from_name(name) + if sym and sym != GDK_VOID_SYMBOL: + return sym if len(name) == 1: return Gdk.unicode_to_keyval(ord(name)) - return 0 + return GDK_KEY_NAMES.get(name.lower(), 0) def keycode(name): @@ -166,11 +176,16 @@ def keycode(name): raise RuntimeError("Gdk unavailable; cannot resolve keycode for: " + name) sym = keysym(name) if not sym: - raise RuntimeError("Unsupported key: " + name) + raise invalid_key("Unsupported key: " + name) keymap = Gdk.Keymap.get_default() ok, entries = keymap.get_entries_for_keyval(sym) if not ok or not entries: - raise RuntimeError("No hardware keycode mapped for key: " + name) + raise OperationFailure( + "No hardware keycode mapped for key: " + name + + ". Choose a key available in the desktop keyboard layout.", + failure_code="unsupported_action", failure_phase="precondition", + input_events_emitted=False, + ) return int(entries[0].keycode) @@ -182,9 +197,9 @@ def parse_key(key): must fail before the window is raised, not after. """ normalized_expression = normalize_key_expression(key) - parts = [part for part in normalized_expression.split("+") if part] - if not parts: - raise RuntimeError("Unsupported key: " + str(key)) + parts = normalized_expression.split("+") + if not all(parts): + raise invalid_key("Unsupported key expression: " + str(key)) main = parts[-1] codes = [] for modifier in parts[:-1]: @@ -193,7 +208,7 @@ def parse_key(key): # Reject unknown modifiers: skipping them would silently turn a # typo such as "ctrl+shft+s" into a bare "s", which is confusing # and very hard to diagnose. - raise RuntimeError( + raise invalid_key( "Unsupported modifier: {}. Use CTRL, SHIFT, ALT, or SUPER; " "Control_L/Ctrl_L and GTK forms such as s are also accepted." .format(modifier)) @@ -218,6 +233,8 @@ def send_key(key): pressed = [] emitted = False primary_error = None + restore_numlock = 0 + symbol = keysym(normalized) try: for code in codes: emitted = emit_keyboard_event( @@ -236,6 +253,21 @@ def send_key(key): "text key {!r}".format(normalized), emitted, ) + elif len(normalized) > 1 and not ( + Gdk.KEY_Shift_L <= symbol <= Gdk.KEY_Hyper_R + or symbol in (Gdk.KEY_Num_Lock, Gdk.KEY_Scroll_Lock)): + restore_numlock = keypad_navigation_numlock_mask(symbol) + if restore_numlock: + emitted = emit_keyboard_event( + restore_numlock, None, Atspi.KeySynthType.UNLOCKMODIFIERS, + "NumLock preparation", emitted) + emitted = emit_keyboard_event( + symbol, + None, + Atspi.KeySynthType.SYM, + "key {!r}".format(normalized), + emitted, + ) else: emitted = emit_keyboard_event( keycode(normalized), @@ -247,6 +279,14 @@ def send_key(key): except Exception as exc: primary_error = exc + if restore_numlock: + try: + emitted = emit_keyboard_event( + restore_numlock, None, Atspi.KeySynthType.LOCKMODIFIERS, + "NumLock restoration", emitted) + except Exception as exc: + if primary_error is None: + primary_error = exc # Release every pressed modifier even if one release fails. Stopping after # the first failed release can leave the remaining modifiers stuck. release_error = None @@ -262,10 +302,10 @@ def send_key(key): except Exception as exc: if release_error is None: release_error = exc - if primary_error is not None: - raise primary_error - if release_error is not None: - raise release_error + for error in (primary_error, release_error): + if error is not None: + error.events_emitted = bool(emitted or getattr(error, "events_emitted", False)) + raise error # Keep the public keyboard-input bound; input method delivery also has a @@ -398,14 +438,16 @@ def find_editable_text(root): character count stays the same, so the text is silently lost and the tool reports success. - The node must therefore be both EDITABLE and FOCUSED, and exactly one - candidate must exist. Without a clear focus owner nothing is guessed + The node must therefore expose Text and be both EDITABLE and FOCUSED, + and exactly one candidate must exist. EditableText is optional: a + keyboard-only editor still needs text readback. Without a clear focus + owner nothing is guessed from visibility or tree order; the caller uses the keyboard path, which is reported as unverified. """ focused = [] for node in iter_all(root): - if not (has_editable_text_iface(node) and has_text_iface(node)): + if not has_text_iface(node): continue if not state_contains(node, Atspi.StateType.EDITABLE): continue diff --git a/computer-use/native/linux/provider/ax_dispatch.py b/computer-use/native/linux/provider/ax_dispatch.py index cb8fd22..f53fff8 100644 --- a/computer-use/native/linux/provider/ax_dispatch.py +++ b/computer-use/native/linux/provider/ax_dispatch.py @@ -21,6 +21,7 @@ from .ax_node_properties import ( parse_text_limit, positive_int, + read_element_text, ) from .x11_windows import ( @@ -298,7 +299,7 @@ def _resolve_target_operation(operation): input_events_emitted=False, ) ( - _element, + element, element_record, _semantic_note, _window, @@ -312,10 +313,14 @@ def _resolve_target_operation(operation): operation.get("known_refs"), operation, ) - return { + response = { "ok": True, "resolvedElement": element_record, } + text_read = (operation.get("observation") or {}).get("text_read") + if text_read is not None: + response["textRead"] = read_element_text(element, text_read) + return response def _perform_ax_action( diff --git a/computer-use/native/linux/provider/ax_dropdown.py b/computer-use/native/linux/provider/ax_dropdown.py index 3b2cecc..007f665 100644 --- a/computer-use/native/linux/provider/ax_dropdown.py +++ b/computer-use/native/linux/provider/ax_dropdown.py @@ -30,6 +30,7 @@ _failure, _quote, _same, + _same_selection_item, _selection_container, _wait_until, text_between, @@ -406,18 +407,128 @@ def _list_selection_first(node): and safe(node.get_selection_iface) is None) +def _select_gtk_option(node, option, wanted): + parent = safe(option.get_parent) + index = safe(option.get_index_in_parent) + + def owned(): + return (parent is not None and index is not None and index >= 0 + and _same_selection_item(safe(parent.get_parent), node) + and _same_selection_item(safe(option.get_parent), parent) + and safe(option.get_index_in_parent) == index + and _same_selection_item( + safe(lambda: parent.get_child_at_index(index)), option)) + + if safe(node.get_selection_iface) is None: + raise _failure("select_unavailable: the element's container does not support " + "selection; no input was emitted.") + if not owned(): + raise _failure("select_unavailable: item ownership changed; " + "no input was emitted.") + + def committed(): + if not owned(): + return False + count = safe(lambda: Atspi.Selection.get_n_selected_children(node)) + selected = (safe(lambda: Atspi.Selection.get_selected_child(node, 0)) + if count == 1 else None) + return (_same_selection_item(selected, option) + and str(safe(selected.get_name, "")).strip() == wanted) + + if committed(): + return wanted + if not all(state_contains(target, state) + for target in (node, option) + for state in (Atspi.StateType.ENABLED, Atspi.StateType.SENSITIVE)): + raise _failure("select_unavailable: the requested option is disabled; " + "no input was emitted.") + action_evidence.update(input_emitted=True, route="selection_api") + try: + Atspi.Selection.select_child(node, int(index)) + except Exception as exc: + raise _failure( + "select_unavailable: the container refused to select the item; " + "the selection may have changed.", code="effect_unknown", + emitted=True, phase="emission") from exc + action_evidence.update(readback_sampled=True) + if not _wait_until(committed, 0.6): + raise _failure( + "select_unverified: the item did not report itself selected.", + code="effect_unknown", emitted=True, phase="readback") + return wanted + + +def is_chromium_select(node): + return (node_role(node) in DROPDOWN_ROLES + and application_toolkit(node) == "chromium" + and (safe(node.get_attributes, {}) or {}).get("tag") == "select") + + +def _select_chromium_option(node, option, wanted): + def committed(): + try: + options = _dropdown_options(node) + matches = [item for item in options if option_name(item) == wanted] + if len(matches) != 1 or not _same_selection_item(matches[0], option): + return None + selected = [] + for item in options: + states = safe(item.get_state_set) + if states is None: + return None + if states.contains(Atspi.StateType.SELECTED): + selected.append(item) + return (any(_same_selection_item(item, option) for item in selected) + and (len(selected) == 1 + or state_contains(node, Atspi.StateType.MULTISELECTABLE))) + except OperationFailure: + return None + + choice = committed() + if choice is None: + raise _failure("select_unavailable: the option identity or selection " + "could not be read; no input was emitted.") + if choice: + return wanted + if node_role(node) == "list box": + raise _failure( + "select_unavailable: this HTML list box has no option action " + "verified to emit both input and change events; no input was emitted.") + if not all(state_contains(target, state) + for target in (node, option) + for state in (Atspi.StateType.ENABLED, Atspi.StateType.SENSITIVE)): + raise _failure("select_unavailable: the requested option is disabled; " + "no input was emitted.") + count = int(safe(option.get_n_actions, 0) or 0) + names = [safe(lambda index=index: option.get_action_name(index), "") + for index in range(count)] + if "select" not in names: + raise _failure("select_unavailable: the option has no select action; " + "no input was emitted.") + action_evidence.update(input_emitted=True, route="semantic_action") + try: + option.do_action(names.index("select")) + except Exception as exc: + raise _failure( + "select_unverified: the option action did not return; " + "the selection may have changed.", code="effect_unknown", + emitted=True, phase="emission") from exc + action_evidence.update(readback_sampled=True) + if not _wait_until(committed, 0.6): + raise _failure( + "select_unverified: the requested option could not be confirmed " + "in its control; the selection may have changed.", + code="effect_unknown", emitted=True, phase="readback") + return wanted + + def select_dropdown_option(node, text, window=None): """Choose the option named ``text`` of a closed or open drop-down. - Browsers expose an HTML