Skip to content

fix(browser): capture a sharp frame when the settled screencast sends none - #628

Open
Robert0622 wants to merge 10 commits into
mainfrom
fix/browser-settled-sharp-frame
Open

Robert0622 wants to merge 10 commits into
mainfrom
fix/browser-settled-sharp-frame

Conversation

@Robert0622

Copy link
Copy Markdown
Collaborator

Problem

TestBrowserScrollEndsOnASharpFrame fails intermittently in the core job. All five recent failures are identical: browser_motion_test.go:78: no frame about 10 s after the wheel event, meaning no frame at all arrived after the scroll. Runs 43479449899, 43459876001, 43431353862, 43391455714 and 43364409673.

markMotion restarts the screencast at half size when a scroll or drag starts, and restarts it at full size 400 ms after input settles. It relied on Chromium re-sending the current page on each restart. Chromium's screencast captures only on repaint and does not always re-send a page that is not changing, so on a static page neither restart produced a frame. attach already works around the same behavior by taking a screenshot for the first frame.

This is also a product issue. When a real scroll ends, the last frames are light; if the full-size restart sends nothing, the panel stays blurry until the page repaints.

Change

  • After the full-size restart, ensureSettledFrame waits up to 300 ms for a frame. If none arrives, and the session, sequence and motion state are unchanged, it captures a screenshot and publishes it as the current frame.
  • TestBrowserScrollEndsOnASharpFrame no longer requires a light frame on a static page, which Chromium does not guarantee. It now requires a full-width frame after the scroll settles.
  • New TestBrowserViewCapturesASharpFrameWhenTheSettledScreencastSendsNone (fake CDP): a screenshot fills in when nothing is sent; a frame that arrives in time is kept and no screenshot is taken.
  • TestBrowserViewOperationsWithFakeCDP cleared view.session without the lock while a content-size read could be running; -race flagged it under load.

Tests

  • The CI failure could not be reproduced locally with Chromium 133 or 156, even pinned to one loaded CPU: Chromium always re-sent here. To check the fix against the CI behavior, the restarts were temporarily changed to only stop the screencast, so Chromium sends nothing:
    • Without ensureSettledFrame, the real-browser test failed exactly like CI (no frame, about 10.5 s).
    • With it, the test passed 10 out of 10 runs.
  • The CI real-browser step command (with coverage flags) passed 3 out of 3 runs.
  • go test ./internal/runtime -run 'TestBrowserView|TestBrowserScrollEndsOnASharpFrame' -count=30 -race passes, and go test ./internal/runtime passes.

… none

Scrolling or dragging in the side browser restarts the screencast at
half size, and 400 ms after input settles it restarts at full size. The
view relied on Chromium re-sending the current page on each restart,
but Chromium captures only on repaint and does not always do that. A
page that stopped changing then left the panel on its last light frame,
and TestBrowserScrollEndsOnASharpFrame failed in CI with "no frame" ten
seconds after the wheel.

When no frame arrives within 300 ms of the full-size restart, capture a
screenshot and publish it, as attach already does for the first frame.
The real-browser test no longer requires a light frame on a static page,
which Chromium does not guarantee; it requires a sharp frame after the
scroll settles. Also take the view lock in a fake-CDP test that cleared
the session while a content-size read could be running.

Co-authored-by: Cursor <cursoragent@cursor.com>
Cover the branches the settled-frame test leaves open: input resuming,
a tab switch or a frame arriving during the wait, a cancelled context,
a failed or empty screenshot, and a frame arriving while the screenshot
is taken (the screenshot must be dropped).

Co-Authored-By: Claude Code <noreply@anthropic.com>

@Yanami-2K Yanami-2K left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

核对了 internal/runtime/browser_view.go 的 ensureSettledFrame:先记下重启后的 seq,等 300ms;调用截图前和写入截图前都在锁内确认 session、seq 未变且不在 motion 中,所以截图期间有真实帧到达、切换页面或又开始滚动时都不会覆盖新帧;截图失败只是不补帧,和原来的行为一致。动机有 5 次 CI 失败的链接作证,且和 attach 用截图补首帧的做法一致;新测试用 fake CDP 断言了“无帧时发布 SHARP 且 seq+1”、“帧及时到达时 0 次截图”以及截图期间有帧到达的情况,core_cases_test.go 也顺手修了一处未加锁的数据竞争。真实浏览器测试不再要求静态页一定出现轻量帧,这部分由已有的 TestBrowserViewCapturesLighterFramesWhileScrolling 覆盖,CI 全绿。合入顺序提示:#510 也在改 browser_view.go(拆分 onEvent),两者改的不是同一个函数,但后合入的一方可能需要 rebase。

Yanami-2K
Yanami-2K previously approved these changes Nov 13, 2026

@Yanami-2K Yanami-2K left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

核对了 internal/runtime/browser_view.go 的 ensureSettledFrame:全尺寸重启 screencast 后等待 browserSettledFrameWait(300ms),只有 session、seq 未变且不在 motion 中才截图,截图返回后又在锁内用 stale() 二次校验再发布,截图失败或空 data 时保持原帧;frameWait 在 time.AfterFunc 外取值,避免与测试改写包级变量竞争。这直接对应描述里 5 次 CI no frame 失败的根因(Chromium 对不重绘的页面重启 screencast 后不一定重发帧),也修了面板停在模糊帧的产品问题,做法与 attach 首帧截图一致。

测试方面,browser_settled_frame_test.go 对发布截图(frame/seq/notify/jpeg 80)、motion/切 tab/已有新帧时不截图、ctx 取消、截图失败、截图期间到帧都有明确断言,browser_test.go 的新用例覆盖端到端的“无帧则截图 / 及时到帧则不截图”;TestBrowserScrollEndsOnASharpFrame 去掉了 Chromium 不保证的“必须出现浅色帧”断言,改为要求 settle 之后的全宽帧(加了 time.Since(scrolled) >= browserMotionSettle,排除飞行中的旧帧),浅色帧行为仍由 fake CDP 的 TestBrowserViewCapturesLighterFramesWhileScrolling 覆盖。core_cases_test.go 给 view.session 写入加锁修掉了 -race 报告。CI 16 项全绿、与 main 无冲突。

非阻塞:browser_settled_frame_test.go patch 被截断,未截断部分及评论中列出的用例已核对;#510 同样改动 browser_view.go(拆分 onEvent),两者改的不是同一函数,后合入的一方注意 rebase。

…sleeps

TestEnsureSettledFrameKeepsTheViewWhenItShouldNotCapture and the second half
of TestBrowserViewCapturesASharpFrameWhenTheSettledScreencastSendsNone changed
the view after a millisecond sleep, so a late goroutine under CPU load raced
ensureSettledFrame and failed the screenshot assertions.

Add a per-view settledFrameSampled hook (nil in production) that runs right
after ensureSettledFrame records the seq it waits on, and change the view or
deliver the resent frame from it.

Co-Authored-By: Claude Code <noreply@anthropic.com>

@Yanami-2K Yanami-2K left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

核对了 browser_view.go 里的 ensureSettledFrame:先在锁内记下当前 seq,等待 browserSettledFrameWait,截图前后两次在锁内检查 session、seq 没变且 !v.motion 才发布截图,避免用旧截图覆盖期间到达的真实帧或新一轮滚动。截图失败时保持原帧,作为兜底是合理的。新增的假 CDP 测试断言了四种情况:收不到帧时补截图且 seq+1 并通知 watcher、按时收到帧时不截图、截图失败、截图期间又收到帧。真浏览器用例改为只要求输入停止后出现全宽帧,和 CI 上 no frame 的根因对得上;core_cases_test.go 也补上了清空 view.session 时缺的锁。CI 全部通过。

小建议(不阻塞合并):browser_test.go 里的 screenshots 闭包和新文件里的 cdpCalls 功能重复,可以合并;截图失败时可以考虑打一条 debug 日志,方便排查。说明:browser_settled_frame_test.go 和 browser_view.go 的 diff 末尾被截断。

Yanami-2K
Yanami-2K previously approved these changes Nov 14, 2026

@Yanami-2K Yanami-2K left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

核对了 browser_view.go 中新增的 ensureSettledFrame:恢复全尺寸截屏后先记下 seq,等待 browserSettledFrameWait,在锁内确认 session、seq 和 motion 都没变才截图,截图返回后再复检一次才发布,避免覆盖期间到达的新帧或新的滚动;ctx 取消和截图失败都直接返回,不会卡住。测试方面,新增的 fake CDP 用例同时断言了“无帧时补截图且只截一次”和“有帧及时到达时不截图”,真实浏览器用例改为要求滚动停止后出现全宽帧,符合 Chromium 不保证重发静态页的实际行为;core_cases_test 的数据竞争修复也正确,CI 全绿。非阻塞建议:截图发布时没有同步更新 frameWidth 等帧尺寸字段,如果下游依赖它做坐标换算,请确认上一张轻量帧的尺寸不会残留;另外 #510 也在重构 browser_view.go 的 onEvent,两边合并时注意 rebase。

@Robert0622

Copy link
Copy Markdown
Collaborator Author

LoopForge PR Review — head a526f7570f288316f06408cc05cb288f7d26a9ad

Verdict: ✅ Pass

  • Scope: 6 files, +332/-20, 3 commits. Fixes the intermittent TestBrowserScrollEndsOnASharpFrame failures (no frame ~10s after scroll; CI runs 43479449899/43459876001/43431353862/43391455714/43364409673): Chromium's screencast only captures on repaint and does not always re-send an unchanged page after markMotion's full-size restart, so the view could stay on a light frame indefinitely.
  • internal/runtime/browser_view.go: new ensureSettledFrame waits browserSettledFrameWait (300ms) after the full-size restart; if session/seq/motion are unchanged (not stale) and no frame arrived, it takes Page.captureScreenshot and publishes it as the current frame. A frame arriving first, input resuming, or a tab switch during the wait correctly skip the screenshot (verified via the stale() closure, always called under v.mu).
  • New internal/runtime/browser_settled_frame_test.go covers the gaps the PR's own self-review flagged: skip on motion/tab-switch/new-frame, ctx cancellation, capture failure/empty data, and screenshot-vs-incoming-frame race — using a settledFrameSampled test-only hook instead of sleeps.
  • Also fixes a pre-existing race in TestBrowserViewOperationsWithFakeCDP (clearing view.session without the lock while a content-size read could be running).
  • TestBrowserScrollEndsOnASharpFrame now asserts a full-width frame after the scroll settles instead of requiring a light frame on a static page (which Chromium doesn't guarantee).

Verification (isolated sandbox, no secrets/production access, Go 1.26.11, no Chromium available here)

  • go build ./...: clean (after adding the gitignored web/native-chat placeholder, same workaround noted on other PRs in this repo).
  • go test ./internal/runtime/... -run 'TestBrowserView|TestEnsureSettledFrame|TestBrowserScrollEndsOnASharpFrame': all PASS; TestBrowserScrollEndsOnASharpFrame SKIPped locally (no Chromium — covered by CI's separate browser job instead).
  • Same filter with -count=10 -race: PASS, confirms the view.session race fix.
  • gofmt -l internal/runtime/ / go vet ./internal/runtime/...: clean.
  • Full go test ./internal/runtime/...: 7 pre-existing failures, reproduced identically on base ca9325a (Docker/Chromium/git-auth unavailable in this sandbox: TestDockerSandboxRunsOnDemandClaudeCodeFromMountedDirectory, TestDockerDiscoveryImagesAndBuildHelpers, TestDockerMasksAndManagerPaths, TestIsolatedRuntimeImageReusesInstalledImage, TestMaterializeSkillBinaryFiles, TestHostInsteadOfUsernamePromptThenAskPass, TestAgentStateMountsAndDockerManagerReclaim) — not a regression from this PR.
  • GitHub CI: 16/16 checks green on this head, including the real-Chromium browser and e2e-browser jobs.

No blocking issues found.

@Robert0622
Robert0622 dismissed stale reviews from Yanami-2K and ghost via b499b91 November 17, 2026 07:42
newexploreor
newexploreor previously approved these changes Nov 17, 2026

@newexploreor newexploreor left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

可以合入,没有阻塞问题。 不过 core 和 browser 两项 CI 还在跑,建议等它们变绿再合。这个 PR 修的就是 core 里偶发失败的用例,这两项的结果最能说明问题。

newexploreor
newexploreor previously approved these changes Nov 17, 2026

@newexploreor newexploreor left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

可以合并,没有阻塞问题
这个 PR 修的是 core 任务里偶发的 no frame 失败:滚动停止后,截屏流重启到全尺寸,但 Chromium 对没有变化的页面不一定重发画面。现在等 300ms 还没收到帧,就主动截一张清晰图补上。这同时修好了产品问题:滚动结束后,侧栏浏览器偶尔会一直停在模糊画面上。

@Robert0622 Robert0622 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

574bbe42 的清晰帧修复通过本地复测。真实 Chromium 连跑两次,滚动结束后都恢复清晰帧。模拟 CDP 的用例也验证了无新帧时补截图、已有新帧时跳过、截图期间收到新帧时保留新帧,以及截图失败时保留原画面。与 main@6a331fb5 试合并时,唯一冲突文件是 CHANGELOG.md。解决该冲突即可重新验证合并状态。

@Robert0622 Robert0622 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审通过 · 4703622d

使用真实 Chromium 连续两轮验证了滚动停止后截取清晰帧、截屏期间有新帧到达、取消和截图失败时保留上一帧。桌面浏览器的静止画面与滚动回归均通过。

已核对本提交的差异和 GitHub 检查,当前没有发现阻塞合并的问题。

@Robert0622 Robert0622 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审通过 · 43604d31

使用真实 Chromium 连续两轮验证了滚动停止后截取清晰帧、截屏期间有新帧到达、取消和截图失败时保留上一帧。桌面浏览器的静止画面与滚动回归均通过。

已核对本提交的差异和 GitHub 检查,当前没有发现阻塞合并的问题。

crispinhu27
crispinhu27 previously approved these changes Nov 21, 2026

@crispinhu27 crispinhu27 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

初审 #628(head 43604d3,merge-base 5fadefc)。

审查范围:full diff(6 文件,+332/-20)。核心变更是 ensureSettledFrame:在锁内、截图前后各检查 session/帧序列未变更且无滚动进行中,防止新帧覆盖更新的帧。

验证:go vet 通过。新增及受影响的 fake-CDP 测试全部通过,10 次迭代。真实浏览器(Chromium 156,prrun 隔离):TestBrowserScrollEndsOnASharpFrame 低负载 6/6 通过;高负载 13 次中 6 次失败,均在本 PR 修改代码之前(Chromium 启动超时或首帧未到),不在 after-scroll 检查处。merge-base 上同一测试复现了 CI 失败的 no frame 问题(1/5 次)。CI 全部 16 项检查通过。

未验证:data race 检测(沙箱无 gcc,CI 不跑 -race)。PR 新增的锁通过代码阅读确认正确。以上不代表代码没有其他缺陷。

@crispinhu27 crispinhu27 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审结论:APPROVE(head e28f67fb,base main@edc27c0)

本次变化:只有一个合并提交 e28f67f,把 main@edc27c0 合进来。

  • --remerge-diff 显示唯一的冲突在 CHANGELOG.md 的 0.4.0 小节。解决时两边都保留:主线的 SOP designer 说明原样保留,本 PR 的 side-browser 说明排在后面。
  • 除 CHANGELOG 外,净 PR diff(edc27c0..e28f67f)和上次批准的 5fadefca..43604d3 逐字节一致(6 个文件,+332/−20)。主线这段区间没有改动 internal/runtime/。

已执行的验证(隔离沙箱用户,不接触任何凭据)

  • go build ./...、go vet ./internal/runtime/ 通过。
  • 用非 root 的 Chromium 156 运行(加了 --no-sandbox 包装):TestBrowserScrollEndsOnASharpFrame、本 PR 新增的 TestBrowserViewCapturesASharpFrameWhenTheSettledScreencastSendsNone、TestEnsureSettledFrame* 等用例 -count=5 全部通过。
  • internal/runtime 全包在 head 和 merge-base 上失败集合完全相同,共 8 个,都依赖 docker/bwrap 环境。TestBrowserViewCapturesLighterFramesWhileScrolling 只在高负载下偶发失败:空闲时 base 和 head 都是 40/40 通过,加压后两边都是 60/60 失败。这个时序问题是原有的,不是本 PR 引入的。
  • GitHub 检查:core、browser、platform-required、e2e-required 等 16 项全部通过。

历史问题:上一轮没有阻塞问题。-race 仍然没跑,因为沙箱里没有 cgo 编译器。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants