Add cached diff-scan polling support to DiffScans.get - #99
Conversation
DiffScans.get now accepts optional query params (cached, omit_unchanged,
omit_license_details) and returns a {"status": "processing", "id": ...}
dict on HTTP 202 instead of logging an error, so clients can poll
GET /orgs/{org}/diff-scans/{id}?cached=true until the computed diff is
ready rather than holding a single idle connection open while the
backend computes (which idle-timeout middleboxes like Azure NAT
gateways kill after ~4 minutes).
Also encode list-valued query params (e.g. committers) as repeated
params in create_from_repo/create_from_ids via urlencode(doseq=True).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
🚀 Preview package published! Install with: pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple socketdev==3.4.0.dev5 |
Eric Hibbs (flowstate)
left a comment
There was a problem hiding this comment.
ai@cursor: Approving — the approach is right, the 202-as-status handling is the correct shape, and the tests cover it. One thing worth hardening before merge, because I think it's live rather than theoretical.
The 202 handler can overwrite the sentinel it documents
result = {"status": "processing", "id": diff_scan_id}
body = response.json()
if isinstance(body, dict):
result.update(body)The merge is unconditional, so any status field in the 202 body wins over "processing":
202 body={'status': 'pending'} -> {'status': 'pending', 'id': 'abc'}
202 body={'id': 'abc'} -> {'status': 'processing', 'id': 'abc'}
202 body=None -> {'status': 'processing', 'id': 'abc'}
That matters because the companion poll loop in SocketDev/socket-python-cli#284 branches on exactly that field:
if response.get("status") != "processing":
scan = response.get("diff_scan") or {}
if scan.get("artifacts") is None:
raise Exception(f"Error fetching diff scan {diff_scan_id}: unexpected response: ...")If a 202 body ever carries a status of its own, the loop exits early, finds no diff_scan in a still-computing response, and raises unexpected response — a hard failure on the final comparison step, which is the symptom this change set exists to remove.
I don't know what the API actually puts in a 202 body, and if it's empty this never fires. But the SDK merges an unknown body defensively while the loop treats anything that isn't "processing" as a finished diff, so the two halves disagree about who owns that field. Cheap to make unconditionally safe by letting the sentinel win:
result = {**body, "status": "processing", "id": diff_scan_id}The doseq=True fix is correct — verified that list values encode as committers=a&committers=b.
|
Eric Hibbs (@flowstate) Good catch. Addressed in e26325a while keeping the hardening in this PR: the SDK now preserves additional 202 response fields but reapplies the canonical I added a regression test with conflicting |
Eric Hibbs (flowstate)
left a comment
There was a problem hiding this comment.
ai@cursor: Verified against e26325a and approving.
Re-ran the repro: the sentinel now wins in every conflicting case, and fields the body adds on top still come through.
202 body={'status': 'pending'} -> {'status': 'processing', 'id': 'abc'}
202 body={'status': 'pending', 'id': 'some-other-id'} -> {'status': 'processing', 'id': 'abc'}
202 body={'queuePosition': 3, 'startedAt': 't0',
'status': 'pending', 'id': 'other'} -> {'queuePosition': 3, 'startedAt': 't0',
'status': 'processing', 'id': 'abc'}
test_diffscans_get_processing_sentinel_wins_unit pins both halves of that — a body contradicting status and id, with an extra retry_after surviving — which is precisely the shape that would have dropped the companion CLI poll loop out early. Suite green at 136 passed / 1 skipped.
DiffScans.getnow accepts optional query params — notablycached=true— and surfaces the API's202 Acceptedprocessing status as{"status": "processing", "id": ...}instead of treating it as an error. For every 202 response, that polling sentinel and the requested diff-scan ID remain authoritative even if the response body contains conflicting values; other response metadata is preserved. This lets clients pollGET /orgs/{org_slug}/diff-scans/{diff_scan_id}?cached=truewith short bounded requests until the computed diff is ready (HTTP 200). Also fixes list-valued query params (e.g.committers) increate_from_repo/create_from_idsto encode as repeated params (urlencode(doseq=True)).Why?
The Python CLI's scan comparison currently uses
fullscans.stream_diff, which holds one HTTP connection open — fully idle — while the backend computes the diff. Network middleboxes with TCP idle timeouts (notably Azure NAT gateways, 4-minute default) kill that connection with a RST, surfacing as intermittentConnectionResetError(104, 'Connection reset by peer')failures on the final comparison step for scans run on self-hosted CI runners.The companion CLI PR switches the comparison to
diffscans.create_from_ids+ pollingdiffscans.get(..., params={"cached": "true"}), which needs this SDK support.omit_license_details/omit_unchangedpassthrough lets the CLI keep the lean-response behavior from CE-224.New unit tests cover the query-string passthrough, the 202 processing status (with an empty, normal, or conflicting response body), and repeated-param encoding.
python -m pytest tests/unit: 136 passed, 1 skipped. Package version staged at 3.4.1.Public Changelog
diffscans.getnow supports query parameters (cached,omit_unchanged,omit_license_details) and returns a stable{"status": "processing", "id": ...}result for HTTP 202, enabling clients to poll for diff-scan results instead of holding a long-lived connection open. The polling sentinel and requested ID remain authoritative if a 202 body contains conflicting values, while other response metadata is preserved. List-valued query params such ascommittersare now encoded correctly indiffscans.create_from_repo/create_from_ids.Refs CE-354