Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,87 @@
# Changelog

## Unreleased

- AX reports when a window snapshot returns only its top-level root.
The diagnostic asks for a screenshot and verification of prior action outcomes
before retrying. It does not infer a renderer crash, reload the application,
or replay an action. Reads with coverage gaps say that omitted content may
still exist; scoped reads do not trigger the window-level diagnostic.

- AX `activate` with a window ref or title can switch the target within an
action batch. Later refs and ref-less steps use the selected window;
refs from other windows require another explicit activation. Each step
reports its window, and the final observation uses the last executed window.
- Linux AX `click` and `scroll` accept `allow_global_input`, defaulting to
`true`. Setting it to `false` forbids driver focus/raise and synthesized
pointer/keyboard input, including editor focus preparation and Qt state
synchronization. Semantic actions can still change application focus.
Unsupported uses reject the whole batch with `invalid_argument` before
input. A required global route returns `global_input_forbidden` without
input from that action. Under this policy, an AT-SPI action returning
false after invocation reports `effect_unknown` without a global retry.
- Linux AX tab clicks prefer the parent container's accessibility selection
operation. Supported list and table items without their own action also
use parent selection. Failed selection verification returns
`effect_unknown` without global input fallback.
- Linux AX scroll can use accessible scrollbar ranges with readback.
A page means five minimum range increments, not pixels; pages remain
approximate, and a clamped boundary no-op is verified. AX results retain
synthesis route notes and warnings without claiming successful effects.
- Linux AX `set_value` reads back the requested control after keyboard fallback.
Unconfirmed values return `effect_unknown` and stop the batch, preserving input
evidence and warning against retyping text that may remain in an editor.

- Linux AX `observe` accepts `read_text` for an existing element `root`.
It pages AT-SPI Text values and complete accessible names or descriptions
in at most 4000 Unicode code points per call. Results include raw text,
offsets, total length, and a next-page call; structured text pages retain
the raw text too. Invalid ranges, owner mismatches, and unavailable Text
fail explicitly. Lower inline budgets reduce page size without raising
tree caps. Each page reads live content; detected length changes fail that
call, but cross-page and same-length edits have no snapshot guarantee.
- Linux AX trees mark clipped fields with their full lengths and read calls.
Non-interactive Text content gets an 80-code-point preview and a bindable
ref. Existing Go cell, find excerpt, name, and description rules remain.
- Text paging and the revised hybrid guide change the model-visible contract
and require a new evaluation attempt. Keep instruction revisions separate;
the interface changes do not establish evaluation performance.
- AX batch errors expose the failed action's reason and a failure-specific
recovery hint while retaining batch progress and input evidence. No-input
argument and unsupported-operation failures use `invalid_argument`;
uncertain input remains `effect_unknown`.
- Invalid AX activation, settle, and double-click calls explain the supported
call shapes without accepting aliases.
- Missing, ambiguous, or moved provider-bound AX observation roots report `stale_ref` and
request a new application observation without the invalid root.

- Linux AX `set_value` and `select` choose Chromium and Electron HTML select
options through accessibility without opening the popup or activating the
window. Both actions read back the selection and fail explicitly when it
cannot be confirmed. HTML list box mutations are refused because their
accessibility actions do not emit both `input` and `change` events.
- Linux AX and GUI accept case-insensitive keypad and X11 key names.
Named keys use their requested symbol, including keypad digits with NumLock
off. Unknown keys report `invalid_argument` with accepted spelling examples
before the key step takes focus or emits input.
- Linux AX preserves references to controls hidden by an in-window modal dialog.
After the dialog closes, each action still checks the live target; a provider
rebuild requires a unique semantic match in the original runtime path and window.
- AX reports text-value evidence for `type` steps without a ref, including
unchanged or unreadable fields. A focus change alone does not confirm typing.
- Linux AX checks focused-field readback after keyboard typing and reports
unconfirmed text explicitly. If window activation changes the resolved text
field, typing stops before keyboard input.
- Linux lists managed application windows that omit `_NET_WM_PID` when
X-Resource can identify their owner. Windows without an accessibility tree
support AX activation without pointer movement. Unverified owners produce
an explicit inventory issue instead of an actionable reference.
- Linux AX `set_value` and `select` choose native GTK combo box options through
accessibility without opening the popup or taking keyboard focus. Both actions
verify the active item and fail explicitly if selection cannot be confirmed.
Combo boxes with an entry use the same option selection; their entry refs
remain available for free-form text editing.

## 1.0.0

First open-source release of Hybrid Computer Use.
Expand Down
27 changes: 14 additions & 13 deletions computer-use/internal/contract/contract_describe.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import (
// tool schemas, the records envelope and the error table below. Adding an
// optional field or operation raises the minor number; removing or changing
// one raises the major number.
const ContractVersion = "1.0"
const ContractVersion = "1.1"

// AXCodeGUIFrameStale is the GUI code for coordinates that refer to a
// screenshot that no longer matches the desktop layout. Nothing was sent.
Expand All @@ -30,18 +30,19 @@ type PublicError struct {
}

var PublicErrorCodes = map[string]PublicError{
AXCodeInvalidArgument: {2, "no", "fix the argument named by path; hint is a corrected call"},
AXCodeWaitTimeout: {3, "maybe", "observe again; completed counts the steps that ran and the message says what the wait saw"},
AXCodeStaleRef: {4, "no", "observe or find again and use the new ref; refs never rebind"},
AXCodeGUIFrameStale: {4, "no", "take a new GUI screenshot and plan coordinates from it"},
AXCodeWindowChanged: {5, "maybe", "observe before retrying; do not repeat the batch blindly"},
AXCodeEffectUnknown: {5, "maybe", "observe before retrying; input may have taken effect"},
AXCodeAppUnavailable: {6, "no", "list apps; the application is not running or not accessible"},
AXCodeAmbiguousWindow: {6, "no", "use a longer title or the wN from apps"},
AXCodeWindowNotFound: {6, "no", "list apps and use a wN or an exact title"},
AXCodeNoFocusedWindow: {6, "no", "activate a window first or pass window_ref"},
AXCodeRuntimeError: {10, "maybe", "retry reads later; observe before retrying input"},
GUICodePermissionDenied: {6, "no", GUIPermissionHint},
AXCodeInvalidArgument: {2, "no", "fix the argument named by path; hint is a corrected call"},
AXCodeWaitTimeout: {3, "maybe", "observe again; completed counts the steps that ran and the message says what the wait saw"},
AXCodeStaleRef: {4, "no", "observe or find again and use the new ref; refs never rebind"},
AXCodeGUIFrameStale: {4, "no", "take a new GUI screenshot and plan coordinates from it"},
AXCodeWindowChanged: {5, "maybe", "observe before retrying; do not repeat the batch blindly"},
AXCodeEffectUnknown: {5, "maybe", "observe before retrying; input may have taken effect"},
AXCodeGlobalInputForbidden: {5, "maybe", "the refused action sent no input; check completed and observe before replanning; keep allow_global_input false unless global input is authorized"},
AXCodeAppUnavailable: {6, "no", "list apps; the application is not running or not accessible"},
AXCodeAmbiguousWindow: {6, "no", "use a longer title or the wN from apps"},
AXCodeWindowNotFound: {6, "no", "list apps and use a wN or an exact title"},
AXCodeNoFocusedWindow: {6, "no", "activate a window first or pass window_ref"},
AXCodeRuntimeError: {10, "maybe", "retry reads later; observe before retrying input"},
GUICodePermissionDenied: {6, "no", GUIPermissionHint},
}

// DescribeContract is the machine-readable contract: what `schema` prints
Expand Down
35 changes: 18 additions & 17 deletions computer-use/internal/contract/published_values.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,23 +24,24 @@ const (
// Public AX error codes. Internal failure codes stay in telemetry; the model
// sees only these.
const (
AXCodeInvalidArgument = "invalid_argument"
AXCodeStaleRef = "stale_ref"
AXCodeWindowChanged = "window_changed"
AXCodeWaitTimeout = "wait_timeout"
AXCodeEffectUnknown = "effect_unknown"
AXCodeAppUnavailable = "app_unavailable"
AXCodeRuntimeError = "runtime_error"
AXCodeAmbiguousWindow = "ambiguous_window"
AXCodeWindowNotFound = "window_not_found"
AXCodeNoFocusedWindow = "no_focused_window"
AXModeApps = "apps"
AXDefaultFindPageSize = 20
AXDefaultWaitTimeout = 5000
AXPublicErrorCodeKey = "public_error_code"
AXPublicErrorPathKey = "public_error_path"
AXPublicErrorHintKey = "public_error_hint"
AXPublicErrorDetailKey = "public_error_message"
AXCodeInvalidArgument = "invalid_argument"
AXCodeStaleRef = "stale_ref"
AXCodeWindowChanged = "window_changed"
AXCodeWaitTimeout = "wait_timeout"
AXCodeEffectUnknown = "effect_unknown"
AXCodeGlobalInputForbidden = "global_input_forbidden"
AXCodeAppUnavailable = "app_unavailable"
AXCodeRuntimeError = "runtime_error"
AXCodeAmbiguousWindow = "ambiguous_window"
AXCodeWindowNotFound = "window_not_found"
AXCodeNoFocusedWindow = "no_focused_window"
AXModeApps = "apps"
AXDefaultFindPageSize = 20
AXDefaultWaitTimeout = 5000
AXPublicErrorCodeKey = "public_error_code"
AXPublicErrorPathKey = "public_error_path"
AXPublicErrorHintKey = "public_error_hint"
AXPublicErrorDetailKey = "public_error_message"
)

var AXActionTypes = []string{
Expand Down
47 changes: 35 additions & 12 deletions computer-use/internal/contract/tool_contract.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ const (
PublicToolGUI = "GUI"
)

const keyChordDescription = "One chord, e.g. [\"CTRL\",\"S\"]. " +
"PAGEUP/PAGE_UP/PAGEDOWN/PAGE_DOWN. Linux named keys are case-insensitive, " +
"including KP_ADD, KP_SUBTRACT, KP_MULTIPLY, KP_DIVIDE, KP_DECIMAL, KP_ENTER, " +
"KP_0 through KP_9 and mapped X11 names. Use PLUS for '+'."

const (
GUIMaxActions = 64
GUIMaxScrollDelta = 2000
Expand All @@ -27,6 +32,7 @@ const (
GUIMaxSettleMS = 3000
AXBatchMaxActions = 64
AXBatchMaxText = 20000
AXMaxReadText = 4000
MaxClickCount = 10
MaxScrollPages = 100
MaxVerifyTimeout = 60000
Expand Down Expand Up @@ -187,7 +193,7 @@ func axInputSchema() map[string]any {
"app": nonBlankString(AppArgumentHelp),
"window_ref": map[string]any{
"type": "string", "pattern": "^w[1-9][0-9]*$",
"description": "observe/find: window to read. act: omit when actions have refs, which name the window; it only sends ref-less keys/text to wN instead of the focused window.",
"description": "observe/find: window to read. act: omit when actions have refs, which name the window; otherwise sets the initial target for ref-less steps instead of the focused window.",
},
"scope": map[string]any{"type": "string", "enum": AXScopeValues, "description": "observe/find; default window."},
"observe": map[string]any{
Expand All @@ -198,6 +204,15 @@ func axInputSchema() map[string]any {
"type": "string", "pattern": "^(focused|r[1-9][0-9]*)$",
"description": "observe: subtree at rN or the focused element.",
},
"read_text": map[string]any{
"type": "object", "additionalProperties": false, "required": []string{"field"},
"description": "Linux observe only: read one live root rN without input or a tree. The ref identifies its window; omit app, scope and observe. Offsets count Unicode code points. Follow next_read to continue; pages are live reads, not a frozen document.",
"properties": map[string]any{
"field": map[string]any{"type": "string", "enum": []string{"name", "value", "description"}, "description": "value reads the node's AT-SPI Text interface; unavailable text returns an error. name and description read those attributes."},
"start": map[string]any{"type": "integer", "minimum": 0, "maximum": 2147483647, "description": "Start offset, inclusive; default 0."},
"limit": map[string]any{"type": "integer", "minimum": 1, "maximum": AXMaxReadText, "description": "Maximum code points; default 4000."},
},
},
"query": map[string]any{"$ref": "#/$defs/query"},
"limit": map[string]any{"type": "integer", "minimum": 1, "maximum": MaxAXFindPageSize, "description": "find page size; default 20."},
"cursor": map[string]any{
Expand All @@ -213,7 +228,7 @@ func axInputSchema() map[string]any {
"type": map[string]any{"type": "string", "enum": AXActionTypes},
"ref": map[string]any{
"type": "string", "pattern": "^[rw][1-9][0-9]*$",
"description": "rN element; wN for activate/close. Omit for keys/text to the focused window.",
"description": "rN element; wN for activate/close. Omit for keys/text to the batch target (focused window by default).",
},
"to_ref": map[string]any{
"type": "string", "pattern": "^r[1-9][0-9]*$",
Expand All @@ -227,16 +242,20 @@ func axInputSchema() map[string]any {
"keys": map[string]any{
"type": "array", "minItems": 1, "maxItems": GUIMaxKeys,
"items": map[string]any{"type": "string", "pattern": "^\\S(.*\\S)?$", "maxLength": GUIMaxKeyLength},
"description": "One chord, e.g. [\"CTRL\",\"S\"].",
"description": keyChordDescription,
},
"value": map[string]any{"type": "string", "maxLength": MaxActionText},
"direction": map[string]any{"type": "string", "enum": []string{"up", "down", "left", "right"}},
"pages": map[string]any{"type": "number", "exclusiveMinimum": 0, "maximum": MaxScrollPages},
"window": nonBlankString("wait: a top-level window whose title contains this, on the whole desktop."),
"query": map[string]any{"$ref": "#/$defs/query"},
"state": nonBlankString("wait: ref state, e.g. checked."),
"gone": map[string]any{"type": "boolean", "description": "wait until it is absent."},
"settle": map[string]any{"type": "boolean", "description": "wait: true waits until the UI changes after the step before it, then stops changing."},
"allow_global_input": map[string]any{
"type": "boolean", "default": true,
"description": "Linux click/scroll only. false forbids driver focus/raise and pointer/keyboard synthesis. Semantic actions may themselves change app focus. Unsupported uses reject the batch before input.",
},
"window": nonBlankString("wait: a top-level window whose title contains this, on the whole desktop."),
"query": map[string]any{"$ref": "#/$defs/query"},
"state": nonBlankString("wait: ref state, e.g. checked."),
"gone": map[string]any{"type": "boolean", "description": "wait until it is absent."},
"settle": map[string]any{"type": "boolean", "description": "wait: true waits until the UI changes after the step before it, then stops changing."},
"timeout_ms": map[string]any{
"type": "integer", "minimum": 1, "maximum": MaxVerifyTimeout,
"description": "wait: default 5000.",
Expand Down Expand Up @@ -269,7 +288,7 @@ func guiActionProperty(includeScreenshot bool) map[string]any {
}
keyArray := map[string]any{
"type": "array",
"description": "PAGEUP/PAGE_UP/PAGEDOWN/PAGE_DOWN.",
"description": keyChordDescription,
"minItems": 1,
"maxItems": GUIMaxKeys,
"items": map[string]any{
Expand Down Expand Up @@ -419,12 +438,16 @@ func GUIInputSchema() map[string]any {

// axToolDescription is model-visible on every client, including ones that
// compact the input schema. Clients may truncate descriptions at 1,000 bytes.
const axToolDescription = "Accessibility for application windows. Modes: " +
const axToolDescription = "Accessibility for application windows. Linux: observe with root rN and read_text reads bounded pages of full name, value (AT-SPI Text), or description; follow next_read. Modes: " +
"apps lists confirmed windows (wN, title, app, active) and reports incomplete inventories; missing windows then remain unknown. observe returns the window tree with " +
"rN refs. find returns matching rN. act runs ordered steps, checks them before input, " +
"stops at the first failed step, and reports each target's change, windows " +
"opened/closed/focused, and a diff. Copy rN/wN exactly; stale refs fail. keypress/type " +
"without ref go to the focused window. select marks ref, its text substring, or ref..to_ref " +
"opened/closed/focused, and a diff. " +
"activate with wN or title switches the batch target for later refs and ref-less steps. " +
"Other-window refs fail until another activate selects their window. " +
"Each step reports its window; final observation uses the last executed window. " +
"Copy rN/wN exactly; stale refs fail. keypress/type " +
"without ref use the batch target, initially the focused window unless refs or window_ref select it. select marks ref, its text substring, or ref..to_ref " +
"(paragraphs, cells) and reports the selection read back. activate/close take a title substring or wN. " +
"A wait step sends no input and fails the batch on timeout: window (title, any app), " +
"query (as find), ref with value/state/text, or text; gone inverts. settle needs no " +
Expand Down
5 changes: 5 additions & 0 deletions computer-use/internal/core/app_snapshot.go
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,8 @@ type windowSelection struct {
}

type ElementRecord struct {
TextTruncated []string `json:"textTruncated,omitempty"`
TextLengths map[string]int `json:"textLengths,omitempty"`
Index int `json:"index"`
RuntimeID []int `json:"runtimeId,omitempty"`
RuntimeIDAliases [][]int `json:"runtimeAliases,omitempty"`
Expand Down Expand Up @@ -287,6 +289,9 @@ func (s *AppSnapshot) renderedText() string {
lines = append(lines, "windows:")
lines = append(lines, renderedWindowInventory(s.Windows)...)
}
if note := s.contentUnavailableNote(); note != "" {
lines = append(lines, note)
}
if normalizedAXScope(s.Scope) == contract.AXScopeContent && s.scopeRoot != nil {
lines = append(lines, fmt.Sprintf(
"scope: content root=%s %q",
Expand Down
Loading
Loading