feat: Add Isolate kit - #291
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughChangesIsolate adds a complete issue-reproduction kit. It includes Lamatic planning, Daytona sandbox execution, evidence certification, MCP and HTTP APIs, a Next.js workbench, documentation, configuration, and tests. Isolate reproduction workflow
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
:robot_face: AgentKit Structural ValidationNew Contributions Detected
Check Results
🎉 All checks passed! This contribution follows the AgentKit structure. |
|
Failure recorded at 2026-07-24T10:48:07Z UTC. If this PR is not fixed within 4 weeks it will be automatically closed. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
kits/isolate/apps/lib/runtime/daytona.ts (1)
344-436: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winBiome's
noUnsafeFinallyis still tripping on lines 433 and 711 — the restructure didn't actually clear it.Both blocks now correctly preserve the primary error/log the cleanup failure per the earlier review's intent, but Biome flags any
throwinside afinally, conditional or not (confirmed: its rule engine doesn't reason about the surroundingif). The static analysis hints for this diff still show[error] 433-433and[error] 711-711for "Unsafe usage of 'throw'", so this will keep failing lint even though the runtime semantics are sound now.Move the "throw only if the body succeeded" decision outside the
finallyblock entirely to satisfy both the linter and the original request.🧹 Move the conditional throw out of `finally`
- } catch (error) { - probeError = error; - throw error; - } finally { + } catch (error) { + probeError = error; + } finally { let cleanupError: unknown; let cleaned = false; for (let attempt = 0; attempt < 2; attempt += 1) { ... } - if (!cleaned) { - if (probeError) { - console.error("Isolate probe cleanup failed after probe failure", cleanupError); - } else { - throw cleanupError; - } - } + if (!cleaned && probeError) { + console.error("Isolate probe cleanup failed after probe failure", cleanupError); + } + this.pendingProbeCleanupError = !cleaned && !probeError ? cleanupError : undefined; } + if (probeError) throw probeError; + if (this.pendingProbeCleanupError) throw this.pendingProbeCleanupError;Apply the analogous restructuring to
resetWorkspace's try/catch/finally around line 675-715.Also applies to: 675-715
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kits/isolate/apps/lib/runtime/daytona.ts` around lines 344 - 436, Restructure the probe cleanup flow so the finally block performs cleanup and records cleanupError but contains no throw; after the try/finally completes, conditionally throw cleanupError only when no probeError occurred, while preserving primary probe errors and cleanup logging. Apply the same pattern to resetWorkspace’s try/catch/finally cleanup flow, ensuring both paths satisfy noUnsafeFinally without changing error precedence.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/isolate/apps/lib/investigate.ts`:
- Around line 63-82: Update assertRequiredLocalServiceStarted to detect
arbitrary repository service-script names instead of only the literal service,
server, and api keys, so local-service commands require the corresponding
package-manager run command. Preserve the existing local-URL and rejection
behavior, and escape the captured serviceScript before interpolating it into new
RegExp because repositoryContext is untrusted.
In `@kits/isolate/apps/lib/runtime/github.ts`:
- Around line 57-60: Update the request timeout handling around AbortSignal.any
to avoid relying on the affected runtime behavior: retain a strong reference to
the composed timeout signal for the request lifetime, or use a runtime-supported
equivalent that cannot be garbage-collected before firing. Preserve
options.signal composition and the 10-second timeout, and verify the deployed
Node.js runtime includes the required GC fix.
In `@kits/isolate/apps/package.json`:
- Around line 15-17: Update the dependency declarations in package.json to use
compatible Next.js 14–15 and React 18 releases, including matching react-dom and
both `@types/react` packages. Then regenerate the lockfile so it reflects the
selected versions and resolved dependency graph.
In `@kits/isolate/prompts/isolate-reproduction-system.md`:
- Line 8: Update the TUI unsaved-exit evidence contract to replace the
unsupported “bun --cwd” wording with the allowed structured workspace form “bun
run --cwd <relative-package-directory> <script>”. Preserve the existing
requirements for working-directory-relative artifact paths and all runtime-owned
fixture, input, assertion, repeat, and cleanup behavior.
---
Outside diff comments:
In `@kits/isolate/apps/lib/runtime/daytona.ts`:
- Around line 344-436: Restructure the probe cleanup flow so the finally block
performs cleanup and records cleanupError but contains no throw; after the
try/finally completes, conditionally throw cleanupError only when no probeError
occurred, while preserving primary probe errors and cleanup logging. Apply the
same pattern to resetWorkspace’s try/catch/finally cleanup flow, ensuring both
paths satisfy noUnsafeFinally without changing error precedence.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 39f9c593-c9b1-4fa4-9812-1fbd0bec986d
📒 Files selected for processing (24)
kits/isolate/README.mdkits/isolate/agent.mdkits/isolate/apps/lib/investigate.tskits/isolate/apps/lib/investigation-request.tskits/isolate/apps/lib/lamatic-planner.tskits/isolate/apps/lib/runtime/claim.tskits/isolate/apps/lib/runtime/daytona.tskits/isolate/apps/lib/runtime/evidence.tskits/isolate/apps/lib/runtime/github.tskits/isolate/apps/lib/runtime/mcp.tskits/isolate/apps/lib/runtime/policy.tskits/isolate/apps/package.jsonkits/isolate/apps/tests/claim.test.tskits/isolate/apps/tests/concurrency.test.tskits/isolate/apps/tests/daytona.test.tskits/isolate/apps/tests/evidence.test.tskits/isolate/apps/tests/investigate.test.tskits/isolate/apps/tests/investigation-report.test.tskits/isolate/apps/tests/investigation-request.test.tskits/isolate/apps/tests/lamatic-planner.test.tskits/isolate/apps/tests/mcp.test.tskits/isolate/apps/tsconfig.jsonkits/isolate/lamatic.config.tskits/isolate/prompts/isolate-reproduction-system.md
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
kits/isolate/apps/lib/investigate.ts (1)
161-164: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winMission update: the "bun --cwd" briefing never got the memo.
This runtime-injected
evidenceGuidancestring still tells the planner "whencommandusesbun --cwd, adjust the path" — the exact wording already corrected tobun run --cwdinkits/isolate/prompts/isolate-reproduction-system.md. Left as-is, a TUI-mode plan following this instruction literally can emit the unsupportedbun --cwdrunner form, which the command policy rejects, costing a plan-repair retry for the wholetui_unsaved_exitevidence path.🕵️ Sync the wording with the fixed prompt file
-Resolve every build-artifact path from the effective working directory of command; when command uses bun --cwd, adjust the path for that directory. +Resolve every build-artifact path from the effective working directory of command; when command uses bun run --cwd, adjust the path for that directory.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kits/isolate/apps/lib/investigate.ts` around lines 161 - 164, Update the `evidenceGuidance` text for `assertion?.kind === "tui_unsaved_exit"` to refer to the supported `bun run --cwd` form instead of `bun --cwd`, while preserving the existing path-adjustment instruction and all other guidance.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@kits/isolate/apps/lib/investigate.ts`:
- Around line 161-164: Update the `evidenceGuidance` text for `assertion?.kind
=== "tui_unsaved_exit"` to refer to the supported `bun run --cwd` form instead
of `bun --cwd`, while preserving the existing path-adjustment instruction and
all other guidance.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 783ba816-1b42-45f9-8a7e-4ace6c7baa69
⛔ Files ignored due to path filters (1)
kits/isolate/apps/bun.lockis excluded by!**/*.lock
📒 Files selected for processing (13)
kits/isolate/apps/lib/investigate.tskits/isolate/apps/lib/runtime/daytona.tskits/isolate/apps/lib/runtime/github.tskits/isolate/apps/lib/runtime/investigation-report.tskits/isolate/apps/next-env.d.tskits/isolate/apps/package.jsonkits/isolate/apps/styles.d.tskits/isolate/apps/tests/investigate.test.tskits/isolate/apps/tests/investigation-report.test.tskits/isolate/apps/tests/lamatic-planner.test.tskits/isolate/apps/tests/plan.test.tskits/isolate/apps/tsconfig.jsonkits/isolate/prompts/isolate-reproduction-system.md
|
@akshatvirmani Done: the PR is ready for final review. All CodeRabbit findings are addressed and resolved, CodeRabbit approved the exact current head, repository validation and the full local test/typecheck/build gate pass, and the production demo is deployed and Ready. |
|
Hi @Dhruv2mars! 👋 Before this PR can be reviewed by maintainers, please resolve all comments and requested changes from the CodeRabbit automated review. Steps to follow:
This helps keep the review process efficient for everyone. Thank you! 🙏 |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
kits/isolate/apps/lib/investigate.ts (1)
280-289: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winKeep the Agent hypothesis field consistent with the remaining investigation paths.
This exploratory evidence review assigns
report.summarytohypothesis, while certified and TUI paths return the planner hypothesis and the UI labelsresult.hypothesisas the preliminary Lamatic hypothesis. Use the planner hypothesis if this field must continue representing the agent’s planner hypothesis; otherwise adjust the field name/contract and update the UI to prevent calling the post-runtime summary an agent hypothesis.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kits/isolate/apps/lib/investigate.ts` around lines 280 - 289, Update the exploratory evidence-review return object in the investigate flow to use the planner’s hypothesis for the `hypothesis` field, matching the certified and TUI paths, instead of `report.summary`. Preserve the existing report summary in its appropriate report field, and keep the `result.hypothesis` UI label consistent with the value’s planner-hypothesis contract.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@kits/isolate/apps/lib/investigate.ts`:
- Around line 280-289: Update the exploratory evidence-review return object in
the investigate flow to use the planner’s hypothesis for the `hypothesis` field,
matching the certified and TUI paths, instead of `report.summary`. Preserve the
existing report summary in its appropriate report field, and keep the
`result.hypothesis` UI label consistent with the value’s planner-hypothesis
contract.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 3aa13f47-e669-4460-ba34-8fcf3a224640
📒 Files selected for processing (2)
kits/isolate/apps/lib/investigate.tskits/isolate/apps/tests/investigate.test.ts
|
/validate |
|
📡 Running Studio validation — results will appear here shortly. |
Studio Runtime Validation (Phase 2)✅ Studio validation passed. The kit loaded successfully in Lamatic Studio. This PR is ready for final review and merge. |
The exploratory path returns the post-runtime evidence-review summary in the hypothesis field while the certified and TUI paths return the planner hypothesis, but the UI labelled every case "Agent hypothesis". Add an explicit hypothesisSource field and label the section from it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Studio runtime validation rejected the flow with "config_json.nodes must be a non-empty array". Every Studio-exported flow in this repository serialises meta/inputs/references/nodes/edges as JSON with quoted keys; this flow used unquoted JavaScript object keys. Re-serialise the flow and model config in the canonical format and rename the externalised prompt and model-config files to the documented <flow>_<node>_<role> convention. The node graph, edges, meta, and inputs are byte-for-byte equivalent after parsing; only serialisation and reference file names changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/isolate/apps/components/investigation-workbench.tsx`:
- Line 24: Update the exploratory determination in the investigation component
to treat hypothesisSource === "evidence_review" as exploratory before applying
the existing verdictOwner === "lamatic" fallback for older responses. Ensure the
verdictOwner type also permits "runtime" when it models the API response, and
preserve the corresponding header and findings behavior.
In `@kits/isolate/flows/isolate-reproduction.ts`:
- Line 115: Update the output mapping for the LLM node so REPORT_MODE preserves
the expected top-level report envelope instead of wrapping the generated
response under plan. Keep the existing plan contract for non-report output, and
ensure parseInvestigationReport() receives the report payload at the expected
top level.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 65bc0c4b-ab70-4d6f-a743-9aa88e82ba5f
📒 Files selected for processing (7)
kits/isolate/apps/components/investigation-workbench.tsxkits/isolate/apps/lib/investigate.tskits/isolate/apps/tests/investigate.test.tskits/isolate/flows/isolate-reproduction.tskits/isolate/model-configs/isolate-reproduction_llmnode-887_generative-model-name.tskits/isolate/prompts/isolate-reproduction_llmnode-887_system.mdkits/isolate/prompts/isolate-reproduction_llmnode-887_user.md
CodeRabbit's pre-merge docstring coverage check reported 1.56% against an 80% threshold. Add JSDoc to every exported function, class, and route handler across lib/, app/api/, and components/, plus the internal helpers that carry the safety-relevant reasoning: the command policy tokenizer, the deadline budget, output redaction, and the certification gate. Documentation only; no behaviour change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The report now carries hypothesisSource so a reader can tell a pre-probe planner hypothesis from a post-probe evidence-review summary. Record that in the README alongside the outcome table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
/validate |
|
📡 Running Studio validation — results will appear here shortly. |
The evidence-review path returns verdictOwner "runtime", so the component's exploratory branch was unreachable: an evidence review was headed "Runtime-certified" and its runs were scored pass/reject against a gate that never ran. Derive exploratory from hypothesisSource, keeping the verdictOwner check for responses that predate that field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@akshatvirmani All CodeRabbit findings are addressed and every review thread is resolved. Studio validation passes. This is ready for review — please take a look. |
2 participants
Isolate
Live demo · Evaluation issue · Kit documentation
Problem
Open-source maintainers regularly receive issues that describe a symptom but omit the exact environment and commands needed to reproduce it. An AI agent can propose a plausible explanation, but its confidence is not evidence.
Isolate turns a public GitHub issue into a repeatable, machine-verifiable reproduction report inside a disposable sandbox.
What this kit adds
Trust boundary
The model investigates. The runtime verifies.
Lamatic forms a hypothesis and proposes a candidate probe plus a nearby negative control. It cannot mark its own work reproduced. The runtime derives one exact assertion from the issue and certifies
reproducedonly when:If that gate does not pass, Isolate returns
not_reproduced_under_tested_conditionsorblocked.Architecture
flowchart LR A[Public GitHub issue] --> B[Deterministic intake] B --> C[Disposable Daytona sandbox] C --> D[Lamatic probe planner] D --> E[Runtime command policy] E --> F[Candidate twice plus control] F --> G[Deterministic evidence gate] G --> H[Markdown and JSON report]Reviewer path
The fixture describes the symptom and observed output without supplying the reproduction command. Isolate must inspect the repository and discover the correct repository-owned CLI invocation.
Verification
Scope
The initial release supports public Node.js, TypeScript, Bun, and terminal/CLI issues. Terminal-output certification uses an exact issue-derived stdout/stderr signature. For unsaved-exit TUI issues, the runtime drives a real PTY, verifies unchanged file state twice, and rejects the claim after a save control. Other issue classes remain blocked without a runtime-owned evidence adapter. Isolate does not edit repository source files, generate fixes, mount repository credentials, push branches, open pull requests, or publish packages.
Challenge checklist
kits/isolateagentkit-challengelabel appliedStatus
Ready for final review. Implementation, submission materials, live demo, and reviewer walkthrough are complete.
Files Added
Configuration and documentation:
kits/isolate/.env.examplekits/isolate/.gitignorekits/isolate/README.mdkits/isolate/agent.mdkits/isolate/constitutions/default.md("Isolate Constitution")kits/isolate/lamatic.config.tskits/isolate/model-configs/isolate-reproduction-model.tskits/isolate/prompts/isolate-reproduction-system.mdkits/isolate/prompts/isolate-reproduction-user.mdLamatic flow:
kits/isolate/flows/isolate-reproduction.ts(Lamatic flow definition)Next.js application core:
kits/isolate/apps/.env.examplekits/isolate/apps/.gitignorekits/isolate/apps/DESIGN.mdkits/isolate/apps/PRODUCT.mdkits/isolate/apps/styles.d.tskits/isolate/apps/next-env.d.tskits/isolate/apps/next.config.tskits/isolate/apps/package.jsonkits/isolate/apps/tsconfig.jsonkits/isolate/apps/app/globals.csskits/isolate/apps/app/layout.tsxkits/isolate/apps/app/page.tsxAPI routes:
kits/isolate/apps/app/api/investigate/route.tskits/isolate/apps/app/api/mcp/route.tsFrontend components:
kits/isolate/apps/components/investigation-workbench.tsxRuntime and service libraries:
kits/isolate/apps/lib/concurrency.tskits/isolate/apps/lib/deadline.tskits/isolate/apps/lib/http-errors.tskits/isolate/apps/lib/investigate.tskits/isolate/apps/lib/investigation-request.tskits/isolate/apps/lib/lamatic-planner.tskits/isolate/apps/lib/runtime/certification.tskits/isolate/apps/lib/runtime/claim.tskits/isolate/apps/lib/runtime/daytona.tskits/isolate/apps/lib/runtime/evidence.tskits/isolate/apps/lib/runtime/github.tskits/isolate/apps/lib/runtime/investigation-report.tskits/isolate/apps/lib/runtime/mcp.tskits/isolate/apps/lib/runtime/plan.tskits/isolate/apps/lib/runtime/policy.tskits/isolate/apps/lib/runtime/probe.tsTest suite:
kits/isolate/apps/tests/certification.test.tskits/isolate/apps/tests/claim.test.tskits/isolate/apps/tests/concurrency.test.tskits/isolate/apps/tests/daytona.test.tskits/isolate/apps/tests/deadline.test.tskits/isolate/apps/tests/evidence.test.tskits/isolate/apps/tests/github.test.tskits/isolate/apps/tests/http-errors.test.tskits/isolate/apps/tests/investigate.test.tskits/isolate/apps/tests/investigation-report.test.tskits/isolate/apps/tests/investigation-request.test.tskits/isolate/apps/tests/lamatic-planner.test.tskits/isolate/apps/tests/mcp.test.tskits/isolate/apps/tests/plan.test.tskits/isolate/apps/tests/probe.test.tsAssets:
kits/isolate/assets/isolate-evidence.jpgLamatic Flow (
kits/isolate/flows/isolate-reproduction.ts)Node types introduced:
triggerNode_1(type:triggerNode, "API Request"): Receivesadvance_schemacontainingissue,repositoryContext,ref, andpolicyFeedbackfields from incoming API request.LLMNode_887(type:dynamicNode, "Generate Text"): Executes system and user prompts (@prompts/isolate-reproduction-system.md,@prompts/isolate-reproduction-user.md) using generative model configuration from@model-configs/isolate-reproduction-model.ts(Gemini 3.1 Flash Lite).responseNode_triggerNode_1(type:responseNode, "API Response"): Maps LLM output to JSON response by embeddingLLMNode_887.output.generatedResponseinto aplanfield withapplication/jsoncontent type.Data flow:
triggerNode_1forwards API request toLLMNode_887via default edge.LLMNode_887processes issue context and produces reproduction plan via LLM inference.responseNode_triggerNode_1receives LLM output and returns structured plan response via default edge.triggerNode_1toresponseNode_triggerNode_1for response routing.High-level behavior:
The flow receives a GitHub issue, repository snapshot, optional commit reference, and policy feedback. The LLM node interprets the issue as untrusted input and produces a structured reproduction plan that specifies either terminal-based candidate/control commands or TUI-based unsaved-exit probe instructions. The runtime then executes this plan in an isolated Daytona sandbox, certifies candidate and control probe evidence against the issue's observed assertion, and reports pass/fail outcome based on the evidence gate criteria (two candidate passes and negative control rejection).