The browser_snapshot tool already supports a boxes parameter that includes element bounding boxes as [box=x,y,width,height] in the accessibility snapshot. However, auto-snapshots taken after action tools (like browser_click, browser_type) don't include bounding boxes.
Use case
We're building a tool (workflow-validation-demo-publisher) that creates interactive product demos from automated browser sessions on Red Hat OpenShift AI. After each click action, we need the bounding box of the clicked element to place hotspots accurately in the demo.
Currently, auto-snapshots only contain the accessibility tree without position data, so we have to use a vision model to estimate coordinates -- which is slow (~5s per frame), expensive, and imprecise (~70-80% accuracy vs pixel-perfect).
Proposed change
Add a snapshot.boxes config option (and a corresponding CLI flag) that, when enabled, includes bounding boxes in all auto-snapshots -- not just explicit browser_snapshot calls.
The change is small since the bounding box infrastructure already exists:
setIncludeFullSnapshot() (used by browser_snapshot) already handles boxessetIncludeSnapshot() (used by action tools) only sets mode, not boxes- The fix is to thread the
boxes config through setIncludeSnapshot() and into the auto-snapshot capture path
Config:
{
"snapshot": {
"mode": "full",
"boxes": true
}
}CLI: npx @playwright/mcp --snapshot-boxes (or similar, happy to follow existing naming conventions)
Estimated scope
~15-20 lines of code + 1 test. I'm happy to submit the PR if assigned.
Context
This would benefit any downstream consumer that needs element coordinates from MCP tool responses -- not just our demo publisher use case. For example, test automation tools that need to verify element positions, or accessibility auditing tools that need layout information alongside the accessibility tree.
The
browser_snapshottool already supports aboxesparameter that includes element bounding boxes as[box=x,y,width,height]in the accessibility snapshot. However, auto-snapshots taken after action tools (likebrowser_click,browser_type) don't include bounding boxes.Use case
We're building a tool (workflow-validation-demo-publisher) that creates interactive product demos from automated browser sessions on Red Hat OpenShift AI. After each click action, we need the bounding box of the clicked element to place hotspots accurately in the demo.
Currently, auto-snapshots only contain the accessibility tree without position data, so we have to use a vision model to estimate coordinates -- which is slow (~5s per frame), expensive, and imprecise (~70-80% accuracy vs pixel-perfect).
Proposed change
Add a
snapshot.boxesconfig option (and a corresponding CLI flag) that, when enabled, includes bounding boxes in all auto-snapshots -- not just explicitbrowser_snapshotcalls.The change is small since the bounding box infrastructure already exists:
setIncludeFullSnapshot()(used bybrowser_snapshot) already handlesboxessetIncludeSnapshot()(used by action tools) only setsmode, notboxesboxesconfig throughsetIncludeSnapshot()and into the auto-snapshot capture pathConfig:
{ "snapshot": { "mode": "full", "boxes": true } }CLI:
npx @playwright/mcp --snapshot-boxes(or similar, happy to follow existing naming conventions)Estimated scope
~15-20 lines of code + 1 test. I'm happy to submit the PR if assigned.
Context
This would benefit any downstream consumer that needs element coordinates from MCP tool responses -- not just our demo publisher use case. For example, test automation tools that need to verify element positions, or accessibility auditing tools that need layout information alongside the accessibility tree.