[spacedouut]

AKA: captured v2.

Summary

Rewrites captured so it captures and encodes in one process. The old design sent raw BGRA over a socket, the agent piped it into ffmpeg, ffmpeg converted it on the CPU and uploaded it back to the GPU. Now the agent spawns captured, frames stay on the GPU where the platform allows, and what comes back is encoded access units. The agent side is agent#3.

Process and protocol

• The agent spawns captured. Control is JSON-RPC 2.0, one message per line, on stdin/stdout. Logs go to stderr. Closing stdin stops it.
• Methods are list, get, set, start, stop, plus a stopped notification. Everything else is a generic property:
  • config: pipeline, display, video.codec, video.encoder, video.fps, video.width, video.height, video.gop, video.preset, video.options
  • live: video.bitrate (in place on NVENC and x264, a session restart elsewhere), video.keyframe
  • read-only: state, video.active_pipeline, video.active_encoder, video.hw, video.size, stats.fps, stats.bitrate, stats.capture_avg_ms, stats.encode_avg_ms, stats.encode_max_ms, stats.dropped, stats.error
  • audio.* returns "audio is not supported yet"
• Media goes to --media fd:3 (an inherited handle on Windows) as records:

  u32 BE len | u8 type | u8 flags | u16 0 | u64 BE pts µs | payload
  type 0 = one access unit (flags bit 0 = keyframe), type 2 = format JSON
  

  A format record comes before each stream config. The writer queue is bounded. When it fills, packets drop until the next keyframe and an IDR gets requested.
• The Unix sockets, the TCP --listen flag and both open security findings go away with them.

Encoding

• av/ is a cgo wrapper over libavcodec, libavutil and libswscale. It handles hw devices and frames (CUDA, VAAPI, D3D11VA, VideoToolbox), a CPU path (BGR0/BGRA straight in, or swscale), and low-latency defaults per encoder. AV1 uses SVT-AV1, since libaom can't keep up in real time.
• pipelines/session.go runs capture and encode on one locked OS thread.
• kms_nvidia zero copy: EGL import of the scanout, a GPU blit (with scaling) into an RGBA8 texture, CUDA-GL interop, cuMemcpy2D into the NVENC frame. Nothing goes through system RAM.
• pipewire_mutter, kms_linear and screencapturekit use the CPU frame path for now.

Measured on observatory (RTX 2070S, DP-1 2560x1440), with cmd/capbench, which now spawns captured and reads the records:

	fps	capture	encode	captured CPU
old: kms_nvidia readback + agent ffmpeg NVENC	60	1.7 ms	in ffmpeg	27% + ffmpeg
readback into in-process NVENC	60	2.5 ms	5.6 ms	29%
zero copy h264_nvenc	60.0	0.55 ms	2.8 ms	6.7%
zero copy hevc_nvenc	60.1			6.3%
libx264 1280x720	60.2			81%
libsvtav1 1280x720	60.1	2.5 ms	12.5 ms	102%
pipewire_mutter into NVENC	50.4	14 ms		46%

Decoded frames show the real desktop with the right colors and orientation, including when scaled to 1080p. Through the agent PR, an owner and a late joiner both got 60 fps, and the late joiner's first frame was a keyframe 71 ms after start.

Not done or not tested

• Windows DXGI to D3D11 encoders and macOS ScreenCaptureKit to VideoToolbox native paths. Both build in CI and use the CPU path.
• VAAPI didn't initialize on observatory (Failed to initialise VAAPI connection), so it's untested.
• No audio capture, no cursor on kms_nvidia.
• SVT-AV1 logs deinit called without sending EOS on shutdown.

Link to Devin session: https://app.devin.ai/sessions/ca77b093f2c242f0a1b0cb198b049b93
Open in Devin Desktop: https://app.devin.ai/desktop/session/ca77b093f2c242f0a1b0cb198b049b93?variant=devin
Requested by: @spacedouut

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[coderabbitai]

Important

• Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 07425718-2ebe-44a0-9453-338f6deb6cfd

Walkthrough

The daemon now selects registered display-capture pipelines, serves JSON-lines control requests and BGRA media frames, and supports Linux KMS, Linux Mutter PipeWire, and macOS ScreenCaptureKit backends. The change adds pooled frame streaming, statistics, and capbench, and removes the former source-selector implementations and synthetic fallback.

Changes: Display Capture Daemon

Layer / File(s)	Summary
Pipeline contract and frame streaming pipelines/pipeline.go, pipelines/stream.go, pipelines/stream_test.go, pipelines/linux/pulled_stream_test.go	The pipeline API adds registration, probing, display selection, pooled frames, and stream statistics. Stream tests cover delivery, dropped frames, cleanup, and startup errors.
Linux KMS capture pipelines/drm_linux.go, pipelines/egl_linux.go, pipelines/kms_*_linux.go, pipelines/linux/*	Linux KMS discovery and framebuffer helpers support linear and NVIDIA capture. The former Linux KMS, GBM, mapping, source-selection, X11, and synthetic-capture implementations and related tests are removed.
Linux Mutter PipeWire capture pipelines/pipewire_mutter_linux.go, pipelines/linux/pipewire.go	The new backend discovers current Mutter monitors, starts a screencast session, and reads BGRA frames through GStreamer. The previous Linux PipeWire implementation is removed.
macOS ScreenCaptureKit capture pipelines/screencapturekit_darwin.go, pipelines/macos/pipeline.go, pipelines/windows/pipeline.go	The ScreenCaptureKit backend lists displays and starts BGRA streams. The previous macOS pipeline and the empty Windows pipeline package file are removed.
Daemon control and media serving main.go, README.md	The daemon selects named or automatically probed pipelines, serves JSON-lines control requests, and streams dimension-prefixed BGRA frames over a media socket. The README documents the pipelines, flags, protocol, and commands.
Capture benchmark cmd/capbench/main.go	The capbench command measures frame timing and throughput, counts duplicate frames, and can report process CPU usage or write the last frame as raw BGRA.

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant capbench
  participant captured
  participant pipelines.Pipeline
  capbench->>captured: Send JSON-lines start request
  captured->>pipelines.Pipeline: Start display stream at requested FPS
  pipelines.Pipeline-->>captured: Return stream and first frame
  captured-->>capbench: Return stream dimensions
  capbench->>captured: Connect to media socket
  captured-->>capbench: Send frame dimensions and BGRA data

Loading

Merge Risk: Moderate · previous review bb9cc

Several capture paths can fail or misbehave in realistic setups. Automatic pipeline selection on NVIDIA machines without the required capability will fail at start instead of falling back, a stalled source can lock up stop and stats requests, KMS capture can return stale frames, and rotated monitors may not negotiate. Resolve these before merging.

Pre-merge checks: 3 · 1 · 1

Failed checks: 1 warning · 1 inconclusive

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 18.42% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 38 functions across 11 files. (1 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check		The title indicates a broad rewrite, but it uses informal and vague wording and does not identify the main technical changes, such as the pipeline registry, native capture pipelines, or removal of syn…	Replace the title with a concise, specific summary of the primary architectural change, such as "Rewrite capture architecture with native pipeline registry and KMS capture".

Passed checks: 3 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 18.42% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 38 functions across 11 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Full details: Title check

Explanation

The title indicates a broad rewrite, but it uses informal and vague wording and does not identify the main technical changes, such as the pipeline registry, native capture pipelines, or removal of synthetic fallback.

Resolution

Replace the title with a concise, specific summary of the primary architectural change, such as "Rewrite capture architecture with native pipeline registry and KMS capture".

Finishing Touches

Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[devin-ai-integration]

🔄 Devin is addressing Devin Review findings

Devin Review found 8 bugs and 1 flag on a95081e. Devin is making a first pass at them and will update this comment once it's done.

Fixed by Devin (8)

• Windows desktop handles leak across streams
• First display blocks all Windows capture
• Page flips can end KMS streams
• Automatic capture chooses unusable NVIDIA pipeline
• Linear capture emits transparent desktop pixels
• Default macOS frame rate reaches ScreenCaptureKit as zero
• Suballocated linear scanouts fail to map
• Oversized frame rate terminates capture daemon

View findings · Follow Devin's work

[devin-ai-integration]

Ran screencapturekit on a macOS 26.6.2 arm64 VM (one 1600x1200 display, 1x, 60 Hz), Go 1.26.10, at 125e5df. go vet ./... && go build && go test ./... pass (and GOOS=linux go vet).

run	fps (capbench)	gap p50 / p99 / max ms	CPU (captured)	grab_avg_ms	stats frames / dropped
static	45.5	20.40 / 34.00 / 41.56	60.5%	17.57	286 / 57
moving (rAF animation in Chrome)	37.8	25.11 / 37.93 / 174.41	60.1%	18.04	279 / 88

capbench's CPU line reads /proc so it prints nothing on macOS. The CPU column is ps cputime over the whole capbench run, as % of one core.

What was broken in screencapturekit_darwin.go:

• stop-stream hung forever. interrupt() called sckit.Stream.Close() from another thread while grab sat in NextFrameBGRA with no deadline. That frees the dylib handle under the waiting call, and its semaphore never fires again, so stream.Close blocked on <-done and capbench never got its stream-stopped. Now grab waits in 100 ms slices and checks a stop flag, and the sckit stream only gets closed in close() on the stream thread. stop, restart and SIGTERM mid-stream all work now.
• -display 0 (capbench's default) failed with display 0 not found, because IDs were raw CGDirectDisplayIDs (1 here). IDs are now indexes, main display first, the same as the Linux pipelines. The CG id goes in name (cg1).
• Retina would have been captured downscaled. sckit's display size and its default stream size are SCDisplay.width/height, which are points. Width/height now come from the CoreGraphics display mode's pixel size (purego), and the stream asks for exactly that with WithResolution. The refresh rate also comes from the mode now. This VM only offers 1x modes, so I couldn't watch it happen on a real Retina screen. Here points = pixels = 1600x1200, and the frames are 1600x1200.

Frames are the real screen, the right way up, with the cursor. Colour swatches read back exactly: (255,0,0), (0,255,0), (0,0,255), (255,255,0), (0,0,0), white 255. The static frame vs screencapture gives PSNR 47.6 dB (the differences are the cursor and the clock).

On a static screen sckit re-delivers the last buffer at about 57/s, so a static stream keeps running at full rate (227/228 identical in the static run) instead of going quiet. The daemon grabs about 57 fps (grab_avg ~17.6 ms), but capbench only receives 35-45. The dropped count shows that gap is lost on the media socket side (7.7 MB frames at ~270-350 MB/s), not in ScreenCaptureKit. I left main.go and stream.go alone.

Screen Recording: nothing to grant. This VM already has it granted to its agent process, and binaries started from its shell inherit it.

Static:

Moving:

Written by Devin

[devin-ai-integration]

Added the Windows dxgi pipeline (pipelines/dxgi_windows.go, e6034c6). No NVIDIA-specific code, so it keeps the plain name.

Tested on Windows Server 2022, 1 output (IddSampleDriver, 1600x1200@60). go vet/build/test pass on windows, and GOOS=linux / GOOS=darwin vet are clean.

dxgi:
  0 DISPLAY2 (Linux FHD) 1600x1200@60.00 +0+0

capbench -secs 5, 60 fps requested:

	fps	gap p50	gap p99	gap max	CPU (1 core)	grab_avg_ms
static	57.0	17.65 ms	20.71 ms	21.66 ms	12%	17.6
motion (full-screen canvas animation)	57.2	17.52 ms	20.62 ms	21.47 ms	23%	17.5

On the static run 285 of 286 frames were identical to the one before, so those are timeout re-sends. On the motion run it was 54 of 287, which is Chrome dropping frames on this GPU-less box.

The first static run only got 39.7 fps (p50 24.4 ms). That's because AcquireNextFrame's timeout rounds to the 15.6 ms system tick. The grabber now holds timeBeginPeriod(1) while it's open, and the numbers above are with that fix.

Dumps converted to PNG are the real desktop and the colors are right:

Not tested here: ACCESS_LOST recovery (I didn't want to trigger UAC or the lock screen on this box), rotated outputs, and multiple monitors.

Written by Devin