Skip to content

Linux AX: input, window and error-handling fixes - #4

Merged
Haoqing Wang (hqhq1025) merged 15 commits into
mainfrom
fix/linux-ax-input-and-errors-20261116
Nov 16, 2026
Merged

Haoqing Wang (hqhq1025) merged 15 commits into
mainfrom
fix/linux-ax-input-and-errors-20261116

Conversation

@hqhq1025

Copy link
Copy Markdown
Collaborator

This branch fixes a set of Linux AX and input problems found while running HCU on real Xfce/AT-SPI desktops with GTK, LibreOffice, Chromium and Electron applications. Each fix is a separate commit with tests.

Input and text

  • type without a ref reports the text effect it had, so a successful entry is no longer hidden behind a focus-only effect line.
  • Spreadsheet set_value through the keyboard fallback is verified on the original control. An uncommitted editor value is reported as unconfirmed instead of as success.
  • read_text on an observe root reads long names, values and descriptions in bounded pages, with an explicit marker and a continuation when text is cut. Body text that only an AT-SPI Text interface exposes is no longer lost.
  • Keypad key names such as KP_ADD keep their case, and a partial input failure keeps its evidence.

Windows and refs

  • Windows without an EWMH process ID (for example Tk) are found through X-Resource instead of being dropped from apps.
  • Refs stay valid across a nested modal dialog in the same application.
  • An activate step inside an act batch switches the batch target. Later steps use the activated window, refs of another window still fail, and the final observation uses the last executed window.
  • click and scroll prefer semantic AT-SPI paths. The new allow_global_input field (Linux, default true) forbids driver focus changes and synthesized pointer or keyboard input; a call that would need them fails before any input is sent.

Selection controls

  • GTK combo boxes and Chromium <select> elements select a value through their own AT-SPI selection, without opening the popup.

Errors and diagnostics

  • Failures keep their cause and a recovery hint, and one public mapping covers all invalid-argument failures.
  • When a window stays alive but no AX content could be read, the snapshot says so and points to a screenshot or a safe reload. It never replays an earlier action.
  • docs/MCP.md documents that Calc handles several Tab keys in one type asynchronously, and how to avoid the problem.

Tests

  • make test: Go tests, the Linux provider runtime tests (1111) and the Python tests pass. Six test_setup_canary tests error in my environment because the docker Python module is missing; they fail the same way on main.
  • The MCP contract golden files are regenerated for the changed tool text.
  • Checked on real desktop containers: two-window activation, semantic scroll and click without focus or pointer change, GTK and Chromium selection, long text in GTK, Chromium and Electron.

Keep the implicit focused editor through keyboard fallback, stop if activation changes it, and verify the exact text insertion with bounded readback. Report unchanged or unavailable text explicitly without retrying it.

Include value evidence for every type step in act reports. Do not treat focus changes or shared batch snapshots as proof of text delivery. Cover rejected, delayed, ambiguous and unreadable input with regression tests.
Resolve missing window owners through bounded X-Resource queries and retain the existing process-generation and window-handle checks. Report unidentified application windows as incomplete instead of silently omitting them. Preserve pointer-free native activation and cover discovery, failure, and identity refresh paths.
Route native GTK combo set_value through the same Selection path as select. Verify the active item's identity and name after one semantic mutation, and never retry GTK selection through popup or keyboard input.

Cover no-op selection, entry-backed controls, refusal, unreadable results and stale ownership. Keep projected LibreOffice editor behavior unchanged.
Resolve case-insensitive named keys from GDK instead of guessing capitalization. Classify unknown keys as invalid_argument and retain spelling guidance through AX batch reporting.

Deliver named symbols with their required keyboard layer, preserve hardware modifier and lock-key behavior, and restore NumLock for keypad navigation. Keep partial-input evidence on synthesis and cleanup failures.

Add keyboard and public-error regressions, document Linux key names, and refresh the AX/GUI contract golden descriptions.
Route provider invalid_argument failures through the shared mapping, so
uncertain input still reports effect_unknown and hints stay callable.
The rejected key spelling remains in the message.
Preflight refs within activation-delimited window segments and carry each selected target through input and final observation. Keep accidental cross-window refs and unexpected window transitions fail-closed.

Route native windows without accessibility trees through the same batch engine. Preserve per-step window evidence, actual final observation identity, wait handoffs, and batching-ablation text.
Select actionless tabs/items through AT-SPI Selection and scroll ancestor ranges with verified readback. Add per-action allow_global_input for Linux click/scroll, reject unsafe fallback before input, and retain synthesis warnings through result compaction and batches.

Update public contract, provider-wire fixtures and recovery guidance. Cover semantic readback, clean refusal, fractional/boundary scrolling, policy serialization, platform validation and partial-batch evidence.
Read back the requested control after keyboard synthesis. Report effect_unknown with input evidence when its value is unconfirmed, stop later batch steps, and warn against retyping text left in an editor. Do not send an implicit commit key.

Cover unchanged, unreadable, normalized, delayed and exact readback in provider tests; cover batch failure propagation despite focus and pixel changes in Go. Document the control-value and application-commit boundary.
Read Text, name, and description pages through fresh ref resolution without desktop input. Preserve compact trees while reporting full lengths, truncation, and continuation calls, including content-only nodes.

Keep MCP, records, SDK, ablations, docs, and wire fixtures aligned. Cover Unicode paging, provider failures, stale refs, response budgets, and display deduplication; verify GTK, Chromium, and Electron reads against an unchanged background sentinel.
Document that ordered key delivery does not guarantee completion of application navigation before following text. Explain Calc's deferred accelerator dispatch and require destination-value inspection after navigation-dependent edits.
@hqhq1025
Haoqing Wang (hqhq1025) merged commit 2f6dbb2 into main Nov 16, 2026
12 checks passed
@hqhq1025
Haoqing Wang (hqhq1025) deleted the fix/linux-ax-input-and-errors-20261116 branch November 16, 2026 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant