diff --git a/LESSONS.md b/LESSONS.md index ca91ae29..77648d9e 100644 --- a/LESSONS.md +++ b/LESSONS.md @@ -669,6 +669,10 @@ Seen 1× while implementing #2179. Reusing old group aggregation would accept ca An interrupted prepared SQL migration may recover its verified import; an active database that disappears must fail closed, not silently revive the old import. Publish a newer-version JSON guard before initialization, validate one authority, and retain a fenced resumable rollback marker across partial moves. A transaction commit must precede Effects acknowledgement without a fallible post-commit read masking that proof. +### 2026-11-03 — A failed acceptance save is not a valid retry result + +Pool acceptance previously left an active claim in memory when saving failed; repeating its idempotency key could return that uncommitted claim. Persist an acceptance marker, roll back failed ownership and require commit proof on replay. Public command replay must similarly bind exact intent/result and current principal while allowing the original successful generation to advance. Keep unknown outcomes explicit; receiving a fact does not grant initiative. + ### 2026-11-03 — Public Chief cannot borrow its approver's identity @@ -755,3 +759,11 @@ Seen 1× (late Task requirement response regression). Stopping on the next sched ### 2026-11-03 — State maps do not encode chronology Seen 1× (actual SQLite native return probe). Two valid Attempts survived, but UUID-key SQL iteration put the successor first and an insertion-order assertion compared it to itself. Use persisted predecessor references for chronology; preserve explicit history positions. Storage order must not decide current execution or evidence. + +### 2026-11-03 — New execution adapters need committed read-model coverage + +Seen 1× (native projection regressions). A correct native stop/binding is insufficient if queries/events still report only SDK workers. Reuse actual activation/exit proofs, distinguish current catalog from frozen definition, hide host details and reference large definitions instead of repeating them. Keep private personal proposals separate from public governance proposals. + +### 2026-11-03 — Read-model lookups must not materialize domain facts + +Seen 1× (actual SQLite API regression). A Task query requested full delivery definitions, creating an unsaved default role catalog. A later Task command persisted it and emitted an unrelated Project change. Return deterministic legacy defaults without mutation and use committed raw catalogs for metadata; keep materialization in explicit writes. diff --git a/PROJECT.md b/PROJECT.md index 14726c16..52340738 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -150,57 +150,9 @@ return/retry/re-review and scripted owner acceptance. Workers/workspaces were re [Evidence](dev/demo/evidence/codex-2026-11-01.json) distinguishes this from the outstanding Claude/mixed-resource/browser/restart criteria and independent repository review. -### Collaboration repairs — 2026-11-01 - -The owner authorized sequential issues [#34–#41](https://github.com/zy95-12/FlyBot/issues/33), one PR each; see [decision](decisions/2026-11-01-collaboration-repair.md). The target is native multi-Bot reuse, per-Bot pools and enforced handoff, restored role contracts, ordinary Chief conversations, safe/watch delegation and configurable Chief inspection schedules. Public hosting is deferred; control-plane stays unchanged. - -R1 (#34) rejects incomplete repository capabilities before workers/workspaces/leases are allocated. Workbench readiness checks Git/runtime/dependency metadata and reports the precise block; this does not claim tests or SDK task delivery have run. Old failed Attempts remain readable. R1 full local verification passed in PR #42; actual shared-service integration remains pending. - -R2 (#35) restores complete Dev/Tester/Reviewer conventions, pinned policy-source hashes and final-execution prompt assembly for native and SDK roles. FlyBot code sealing requires memory/tests/architecture evidence; candidate approval explicitly cannot mint a signed forge PASS. Focused tests cover full rules reaching SDK input, required-source failures and tampering. R2 full verification and later backend/dogfood acceptance are tracked separately. - - -R3 (#36) implements per-logical-Bot pools on existing Task records, atomic claims, -versioned receipts and enforced handoff admission. New shared deliveries default -to dynamic pool dispatch; the legacy DAG is explicit opt-in. Only the receiving -role allocates an execution Session/resource. Candidate returns retain the Task -and create a new Attempt; overall completion waits for Chief judgement. Unit and -SDK contract-process tests cover non-DAG return/rework with one resource. Full -verification, conversation bridging, Chief arbitration/schedules and real shared -service dogfood remain separately tracked acceptance, not implied by these tests. - - -R4 (#37) adds conversation-only private → Chief → original-chat handoff, isolated -from private memory and other members. Query turns cannot publish or execute a -Task; scoped GitHub reads provide actual repository facts. Impactful Task kinds -persist across proposal publication; workbench status separates publication from -queue/start/execution and Chief wait. Targeted privacy, native/SDK-query, no-Task -and credential/revocation tests pass. Full verification and real shared-service -acceptance remain separate; control-plane and public hosting are unchanged. - - -R5 (#38) implements safe/private publication proposals and persistent, revocable -watch grants bound to the current human and Project. Browser logout does not -revoke watch authority; account/membership/grant changes do. Scoped actions use -current human rights and retain provenance. The unchanged SQLite platform can -supply fresh account status through a FlyBot-owned read-only operator helper, -without storing browser cookies or admin credentials. Regression tests cover -logout, restart, current rights, revocation, forged status evidence and fresh WAL -reads. Shared-service provisioning and real-model acceptance are still pending. - -R6 (#39) adds evidence-gated Chief arbitration, audited intervention and configurable -inspection on original DSH scheduled tasks. The JSON template defaults to 30 minutes; -content, cadence, thresholds and pause/enable are editable. Current configured human -authority and generation fences apply to each effect. Unknown process exits retain -capacity; a verified systemd invocation boundary permits local restart recovery. -Deterministic protocol tests return incomplete coverage to Dev then close the same -Task after a fresh Attempt. Real shared-service Codex/timer acceptance remains #41. - -Real shared-service R6 probes passed an actual Codex Chief ordinary inquiry with -zero Tasks and an original DSH 60-second wake after browser logout, then pause/ -resume and restoration to 30 minutes. R7 actual Dev requested Chief arbitration; -Chief returned the same Task to Dev with a fresh Attempt. This exposed a concurrent -resource-release reconciliation race, fixed with a red/green regression. Actual -full R7 delivery and persistent 30-minute/restart acceptance remain in progress. +### Collaboration contract and earlier validation + +Native multi-Bot UI, per-Bot handoff pools and complete pinned role conventions remain required. Ordinary Chief questions are conversations; safe/watch delegation uses current human permissions. Chief inspection uses the original DSH scheduled-task service with editable cadence/content/thresholds (the current template defaults to30 minutes), never a hardcoded framework workflow. Code changes retain required memory/tests/architecture evidence. Earlier R1–R7 implementation and runtime validation are [preserved verbatim](docs/history/2026-11-03-collaboration-repair-evidence.md). Current acceptance follows the convergence records above; no new deployment, Control Plane change or interruption of healthy frontend work is authorized. ### Collaboration execution invariants and historical evidence @@ -323,6 +275,12 @@ New shared Tasks have Chief admission and versioned requirements. Existing Task/ Opt-in SQL retains the original Store/services and saves domain facts plus Effects atomically. JSON becomes a newer-version migration guard, with verified import/recovery and offline export/rollback; no dual writable snapshots. Histories are normalized and large immutable payloads hash-referenced. Routine state updates/views make no additional model calls or Project-position changes. Full-state hydration/scanning remains; storage growth is distinct from #2177 context budgets. No live migration or provider-loop claim. +### 2026-11-03 — Service contracts (#2185) + +The original application service adds private principal-scoped command receipts, bounded Task/Attempt sources, committed snapshots/events and authenticated resumable SSE. Unknown command outcomes are reconciled explicitly, never blindly rerun. Independent personal-agent identity survives carrier/transport rotation. Role acceptance commits before ownership can be replayed. Event receipt grants no role authority; private chat/auth/receipt writes do not affect public Project streams. Existing UI remains; #2186 compatibility and #2189 real deployment/README follow. + Composition checkpoint: current role governance/native exit, bounded context and canonical Task waits now share one execution chain. Native input fences logical role/physical worker separately; Task requirement revision changes stop the old Attempt through existing pool release. Actual composed/native acceptance and remaining API/storage compatibility are still under verification, not deployment. SQLite composition now has an actual DSH native return probe: canonical Chief intake, persisted activation/exit, original role context and single-resource handoff are read from committed SQL snapshots. Inference remains deterministic; no production migration or full-provider workflow acceptance is claimed. + +Service composition adds scoped native execution/exit and current-vs-pinned role metadata, role governance aliases, and bounded per-entity committed snapshots. New projections and final six-case DSH/SQLite validation are under current review; none of this branch is deployed. diff --git a/ROADMAP.md b/ROADMAP.md index d1e2b6d4..fde2a34b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -450,6 +450,8 @@ New shared Tasks have Chief admission and versioned requirements. Existing Task/ #2182 implements opt-in transactional SQLite through the existing Store, exact-identity migration, process-death commit tests and recoverable offline rollback. Retain configured Chief semantic decisions while routine facts/queries remain deterministic. Public scoped event/receipt service #2185, legacy entrypoint cutover #2186 and final real deployment/README #2189 remain subsequent stages; current production state is unchanged. +#2185 unifies command recovery, bounded history/source queries and committed Project event/SSE transport over existing services. Verify persistence failure/restart, current permissions and source hashes separately from model development. Complete legacy interface/cutover compatibility #2186 and original role/native-session dependencies before the final #2189 deployment/README rewrite. + Native prior-host reconciliation now reuses verified supervisor process-group facts; native v2 registered-work/lineage and shared adapter-exit regressions guard against premature initiative transfer. In-flight current-rights abort and composed-stack acceptance remain active. Canonical ongoing role/resource authority checks now reuse native step/SDK input and the existing scheduler + pool stop. Frozen role context remains stable; current revoked rights stop execution, with unknown exit held. Compose descendants and verify parent Task resume/API projections next. diff --git a/changelog.d/2185.added.md b/changelog.d/2185.added.md new file mode 100644 index 00000000..3164a502 --- /dev/null +++ b/changelog.d/2185.added.md @@ -0,0 +1 @@ +Add durable principal-scoped command receipts, bounded Task/Attempt history sources and committed Project snapshots/events with authenticated resumable SSE; fence failed Handoff acceptance and preserve private-agent boundaries. Compose native activation/exit and role-governance facts into bounded queries/events, retain exact frozen-definition references, and expose existing governance through scoped v1 aliases with receipt replay. Keep implicit default catalogs read-only so ordinary queries cannot create unrelated Project changes on a later write. diff --git a/core/src/host/project-agents.js b/core/src/host/project-agents.js index 193667be..d47045ef 100644 --- a/core/src/host/project-agents.js +++ b/core/src/host/project-agents.js @@ -109,8 +109,8 @@ export function createProjectAgents(rt) { rt.requireProjectAccess(authority.actor, authority.agent.projectId) return authority } - function storedDefinitionSet(projectId) { - const existing = ownEntry(definitions(), projectId) + function storedDefinitionSet(projectId, materialize = true) { + const existing = ownEntry(state.projectAgentDefinitions ?? {}, projectId) if (existing) return existing const model = rt.config.projectModel ?? rt.cachedDefault?.() ?? '' const workerNodes = deliveryGraph(model).nodes @@ -118,10 +118,10 @@ export function createProjectAgents(rt) { ...workerNodes.map(node => ({ role: node.id, name: node.name, instructions: node.prompt, model: node.model, skills: node.id === 'developer' ? ['code-change'] : node.id === 'tester' ? ['test-evidence'] : ['commit-review'], tools: [] }))] const record = { projectId, revision: 1, roles: rows.map(row => ({ ...row, id: `${projectId}:${row.role}`, memoryRefs: [], execution: { harness: 'flybot', resourceIds: [] }, toolPolicy: row.role === 'chief' ? { kind: 'scoped-project', filesystem: false, connectors: [] } : { kind: 'registered-worktree', shell: true, connectors: [], isolation: 'cwd-only' } })) } - setEntry(definitions(), projectId, record) + if (materialize) setEntry(definitions(), projectId, record) return record } - function definitionSet(projectId) { return roleGovernance ? roleGovernance.visibleSet(projectId, storedDefinitionSet(projectId)) : storedDefinitionSet(projectId) } + function definitionSet(projectId) { return roleGovernance ? roleGovernance.visibleSet(projectId, storedDefinitionSet(projectId, false)) : storedDefinitionSet(projectId, false) } async function prepareDefinition(role, body, projectId) { if (body.tools !== undefined && !Array.isArray(body.tools)) throw refuse('Invalid role tools') if (Array.isArray(body.tools) && body.tools.length) throw refuse('Agent connector tools need explicitly delegated credentials') @@ -150,7 +150,7 @@ export function createProjectAgents(rt) { rt.projectResources?.checkPool(projectId, next.execution) return next } - roleGovernance = createProjectRoleGovernance(rt, { definitionSet: storedDefinitionSet, prepareDefinition, + roleGovernance = createProjectRoleGovernance(rt, { definitionSet: storedDefinitionSet, readDefinitionSet: projectId => storedDefinitionSet(projectId, false), prepareDefinition, baseDefinition(projectId, role) { const active = definitionSet(projectId).roles.find(row => row.role === role) if (active) return structuredClone(active) @@ -196,6 +196,8 @@ export function createProjectAgents(rt) { return known } if (project.status !== 'active') throw refuse('Project is archived; a new agent cannot be created') + // Creating an actual carrier is a write; metadata reads only peek defaults. + storedDefinitionSet(projectId) let bot if (visibility === 'public' && !room) { const previous = project.chief diff --git a/core/src/host/project-command-receipts.js b/core/src/host/project-command-receipts.js new file mode 100644 index 00000000..07bb9843 --- /dev/null +++ b/core/src/host/project-command-receipts.js @@ -0,0 +1,165 @@ +// Public command recovery over original handlers. Unknown execution outcomes are +// retained, never blindly redispatched. Role acceptance has its own scoped proof. +import { AsyncLocalStorage } from 'node:async_hooks' +import { createHash } from 'node:crypto' +import { constants, closeSync, existsSync, fstatSync, lstatSync, openSync, readSync } from 'node:fs' +import { join } from 'node:path' +import { writeDurableStateFile } from './project-state-storage.js' +import { sourcePage } from './project-service-views.js' +const hash = value => createHash('sha256').update(typeof value === 'string' ? value : JSON.stringify(value)).digest('hex') +const stable = value => Array.isArray(value) ? value.map(stable) : value && typeof value === 'object' ? Object.fromEntries(Object.keys(value).sort().map(key => [key, stable(value[key])])) : value +const fail = (message, code = 'bot/conflict', details = {}) => Object.assign(new Error(message), { code, details }) +export function commandPrincipal(actor, state) { + const carrier = actor.watchGrantId && state?.projectAgents?.[actor.agentId] + const agentId = actor.personalAgentCredentialId ? actor.agentId : carrier?.personalAgentId ?? actor.agentId ?? null + return hash([actor.memberId, agentId, actor.watchGrantId && !carrier?.personalAgentId ? actor.watchGrantId : null]) +} +export function commandEffectVisible(state, effect, actor) { + const id = effect.payload?.commandReceiptId + if (!id) return true + const receipt = Object.hasOwn(state.projectCommandReceipts ?? {}, id) ? state.projectCommandReceipts[id] : null + return Boolean(receipt && receipt.projectId === effect.projectId && receipt.principal === commandPrincipal(actor, state)) +} +export function createCommandReceipts(rt) { + const pending = new Map(), context = new AsyncLocalStorage() + const table = () => (rt.state.projectCommandReceipts ??= {}) + const identity = (projectId, name, requestId, actor) => `command-${hash([projectId, commandPrincipal(actor, rt.state), name, requestId])}` + const rows = () => rt.state.projectCommandReceipts ?? {} + const lookup = id => Object.hasOwn(rows(), id) ? rows()[id] : undefined + function resultCommitted(row) { + const effect = row.resultEffect && rt.state.projectEffects?.[row.resultEffect.id] + return Boolean(rt.projectEffects.intentCommitted(row.resultEffect) && effect?.payload.commandReceiptId === row.id && effect.payload.phase === 'result' && effect.payload.resultSha256 === row.resultSha256) + } + const summary = row => { + const committed = row.status === 'completed' ? resultCommitted(row) : rt.projectEffects.intentCommitted(row.beginEffect ?? row.intentEffect) + const status = row.status === 'completed' && !committed ? 'completion-pending-commit' : row.status + return { version: row.version, id: row.id, projectId: row.projectId, requestId: row.requestId, name: row.name, status, committed, origin: structuredClone(row.origin), createdAt: row.createdAt, updatedAt: row.updatedAt, + resultSha256: row.resultSha256 ?? null, resultBytes: row.resultBytes ?? null, resultRef: row.resultSha256 ? { query: 'command.result', commandName: row.name, commandRequestId: row.requestId, expectedSha256: row.resultSha256 } : null, + outcome: ['running', 'unknown'].includes(status) ? 'unknown-until-reconciled' : status, nextAction: ['running', 'unknown'].includes(status) ? 'query-target-or-Chief' : status === 'completion-pending-commit' ? 'retry-command' : null } + } + rt.projectEffects.register('service.command', { run: effect => ({ commandReceiptId: effect.payload.commandReceiptId, phase: effect.payload.phase, status: 'durable-command-marker' }) }) + function commandOrigin(spec) { + const actor = spec.actor, carrier = actor.watchGrantId && rt.state.projectAgents?.[actor.agentId] + const agentId = actor.personalAgentCredentialId ? actor.agentId : carrier?.personalAgentId + return { kind: actor.chiefJobId ? 'project-chief' : agentId ? 'personal-agent' : actor.watchGrantId || actor.agentId ? 'delegated-bot' : 'human', memberId: actor.memberId, + ...(agentId ? { agentId } : {}), ...(actor.chiefJobId ? { botId: `${spec.projectId}:chief` } : {}) } + } + async function authorize(spec) { + if (rt.state.members?.[spec.actor.memberId]?.status === 'disabled') throw fail('Command principal was disabled', 'bot/forbidden') + if (rt.validateHumanActor && !await rt.validateHumanActor(spec.actor)) throw fail('Command principal was revoked', 'bot/forbidden') + if (!rt.platformIdentityEnabled && !spec.actor.agentId && !spec.actor.watchGrantId && !spec.actor.chiefJobId && spec.actor.sessionId) { + const session = rt.state.authSessions?.[spec.actor.sessionId] + if (!session || session.memberId !== spec.actor.memberId || session.revokedAt) throw fail('Command session was revoked', 'bot/unauthorized') + } + const task = spec.taskId && rt.state.tasks?.[spec.taskId] + rt.requireProjectAccess(spec.actor, spec.projectId, spec.action ?? 'read', task) + } + async function commit(row, phase) { + const token = rt.projectEffects.stage({ kind: 'service.command', key: `${row.id}:${phase}`, projectId: row.projectId, + payload: { commandReceiptId: row.id, phase, intentSha256: row.intentSha256, ...(phase === 'result' ? { resultSha256: row.resultSha256 } : {}) } }) + row[`${phase}Effect`] = token + try { await rt.projectEffects.persist(token) } + catch (error) { rt.projectEffects.discardUncommitted(token); throw error } + } + function resultOf(row) { + let text = row.resultText + if (row.resultFile) { + if (!/^[a-f0-9]{64}$/.test(row.resultSha256)) throw fail('Invalid command result reference') + const file = join(rt.home, 'project-command-results', row.resultSha256 + '.json') + if (!existsSync(file)) throw fail('Command result is unavailable', 'bot/outcome-unknown', { nextAction: 'query-target-or-Chief' }) + const info = lstatSync(file) + if (!info.isFile() || info.isSymbolicLink() || info.size > 8 * 1024 * 1024) throw fail('Command result must be a bounded regular file', 'bot/outcome-unknown') + const fd = openSync(file, constants.O_RDONLY | (constants.O_NOFOLLOW ?? 0)) + try { + const current = fstatSync(fd) + if (current.ino !== info.ino || current.dev !== info.dev || current.size !== info.size) throw fail('Command result file changed', 'bot/outcome-unknown') + const buffer = Buffer.alloc(info.size + 1), size = readSync(fd, buffer, 0, buffer.length, 0) + if (size !== info.size) throw fail('Command result file changed during read', 'bot/outcome-unknown') + text = buffer.subarray(0, size).toString('utf8') + } finally { closeSync(fd) } + } + if (typeof text !== 'string' || Buffer.byteLength(text) > 8 * 1024 * 1024 || hash(text) !== row.resultSha256) throw fail('Command result hash changed', 'bot/outcome-unknown', { nextAction: 'query-target-or-Chief' }) + return JSON.parse(text) + } + async function complete(row, result) { + const text = JSON.stringify(result ?? null), bytes = Buffer.byteLength(text) + if (bytes > 8 * 1024 * 1024) throw fail('Command result exceeds its recovery bound', 'bot/outcome-unknown', { nextAction: 'query-target-or-Chief' }) + row.resultSha256 = hash(text); row.resultBytes = bytes + if (bytes > 65536) { writeDurableStateFile(join(rt.home, 'project-command-results', row.resultSha256 + '.json'), text, { exclusive: true }); row.resultFile = true } + else row.resultText = text + row.status = 'completed'; row.updatedAt = Date.now() + await commit(row, 'result') + } + async function run(spec, invoke, check) { + await authorize(spec) + const id = identity(spec.projectId, spec.name, spec.requestId, spec.actor), intentSha256 = hash(stable({ name: spec.name, args: spec.args, expectedGeneration: spec.expectedGeneration, expectedCandidateSha: spec.expectedCandidateSha })) + const prior = lookup(id) + if (prior && (prior.version !== 'project-command/v1' || prior.intentSha256 !== intentSha256)) throw fail('Command requestId was reused with different content', 'bot/idempotency-conflict') + if (pending.has(id)) { const result = await pending.get(id); await authorize(spec); return result } + const operation = (async () => { + let row = prior + if (row?.status === 'completed') { + if (!resultCommitted(row)) await commit(row, 'result') + await authorize(spec) + return { result: resultOf(row), receipt: summary(row), replayed: true } + } + if (row && ['running', 'unknown'].includes(row.status)) { + const recovered = await spec.reconcile?.(row) + if (!recovered) throw fail('Previous command outcome needs reconciliation; it was not re-executed', 'bot/outcome-unknown', { receiptId: row.id, nextAction: 'command.receipt', targetAction: 'query-target-or-Chief' }) + await complete(row, recovered.result) + await authorize(spec) + return { result: resultOf(row), receipt: summary(row), replayed: true, reconstructed: true } + } + if (row?.status === 'rejected') throw fail('Command was rejected; use a new requestId for changed preconditions', row.failure?.code ?? 'bot/conflict', { nextAction: 'new-command' }) + check?.() + if (!row) { + row = { id, version: 'project-command/v1', projectId: spec.projectId, requestId: spec.requestId, name: spec.name, principal: commandPrincipal(spec.actor, rt.state), intentSha256, + origin: commandOrigin(spec), status: 'prepared', createdAt: Date.now(), updatedAt: Date.now() } + Object.defineProperty(table(), id, { value: row, enumerable: true, writable: true, configurable: true }) + try { await commit(row, 'intent') } + catch (error) { if (!rt.projectEffects.intentCommitted(row.intentEffect) && lookup(id) === row) delete table()[id]; throw error } + } + if (!rt.projectEffects.intentCommitted(row.intentEffect)) await commit(row, 'intent') + row.status = 'running'; row.updatedAt = Date.now() + try { await commit(row, 'begin') } + catch (error) { if (!rt.projectEffects.intentCommitted(row.beginEffect)) row.status = 'prepared'; throw error } + let invoked = false + try { + await authorize(spec); check?.(); invoked = true + const result = await context.run({ commandReceiptId: id, projectId: spec.projectId, name: spec.name, origin: row.origin, taskStamped: false }, invoke) + await complete(row, result) + await authorize(spec) + return { result: resultOf(row), receipt: summary(row), replayed: false } + } catch (error) { + if (row.status !== 'completed') { + row.status = invoked ? 'unknown' : 'rejected'; row.updatedAt = Date.now() + row.failure = { code: typeof error.code === 'string' && error.code.startsWith('bot/') ? error.code : 'bot/command-failed' } + // Retain ambiguous outcome even if recording it fails. The durable begin + // marker still fences automatic replay after a restart. + try { await commit(row, 'failure') } catch { /* Existing begin intent remains the conservative recovery fact. */ } + } + throw error + } + })() + pending.set(id, operation) + try { return await operation } finally { if (pending.get(id) === operation) pending.delete(id) } + } + function get(projectId, requestId, actor, commandName) { + rt.requireProjectAccess(actor, projectId) + if (typeof requestId !== 'string' || !/^[A-Za-z0-9_-]{1,100}$/.test(requestId)) throw fail('Choose a stable command requestId', 'bot/invalid') + const matches = commandName === undefined ? Object.values(rows()).filter(row => row.projectId === projectId && row.principal === commandPrincipal(actor, rt.state) && row.requestId === requestId) : [lookup(identity(projectId, commandName, requestId, actor))].filter(Boolean) + if (matches.length > 1) throw fail('Choose commandName for this reused operation-scoped requestId', 'bot/ambiguous-command', { nextAction: 'specify-command-name' }) + const row = matches[0] + if (!row) throw fail('Command receipt is unavailable to this principal', 'bot/not-found') + return row + } + return { run, source(projectId) { + const current = context.getStore() + if (!current || current.projectId !== projectId || !['task.propose', 'proposal.publish'].includes(current.name) || current.taskStamped) return null + current.taskStamped = true + return { commandReceiptId: current.commandReceiptId, projectId, origin: structuredClone(current.origin) } + }, + get: (projectId, requestId, actor, commandName) => summary(get(projectId, requestId, actor, commandName)), + result(projectId, body, actor) { const row = get(projectId, body.commandRequestId, actor, body.commandName); if (row.status !== 'completed' || !resultCommitted(row)) throw fail('Command result is not committed', 'bot/outcome-unknown'); return { receiptId: row.id, ...sourcePage(resultOf(row), body, hash) } }, + } +} diff --git a/core/src/host/project-development-service.js b/core/src/host/project-development-service.js index e5c04918..9a8c29ca 100644 --- a/core/src/host/project-development-service.js +++ b/core/src/host/project-development-service.js @@ -1,5 +1,7 @@ // One application boundary over existing Project services; no second executor/store. import { ownEntry } from './store.js' +import { createCommandReceipts } from './project-command-receipts.js' +import { executionMetadataView, taskContractView, taskProvenance, taskDetailView, attemptMetadataView, taskHistorySource, sourcePage } from './project-service-views.js' import { createHash } from 'node:crypto' import { taskOutcome } from './project-task-lifecycle.js' import { inspectionFacts } from './project-chief-control.js' @@ -13,8 +15,9 @@ const sameTask = (row, task) => Boolean(row && row.projectId === task.projectId const roles = new Set(['chief', 'developer', 'tester', 'reviewer']) const operations = Object.freeze({ 'consultation.reply': 'project-consultation-reply', 'consultation.ask': 'project-consultation-ask', 'consultation.get': 'project-consultation-get', 'consultation.list': 'project-consultation-list', + 'roles': 'project-roles', 'role.proposals': 'project-role-proposals', 'role.decide': 'project-role-decision', 'handoff.get': 'project-handoff-get', 'handoff.list': 'project-handoff-list', - 'task.change.request': 'project-task-change-request', 'task.run': 'project-task-run', 'task.detail': 'project-task-detail', + 'task.propose': 'task-create', 'task.change.request': 'project-task-change-request', 'task.run': 'project-task-run', 'task.detail': 'project-task-detail', 'task.accept': 'project-task-accept', 'task.reject': 'project-task-reject', 'attempt.stop': 'project-attempt-stop', 'attempt.cleanup': 'project-attempt-cleanup', 'attempt.integration-state': 'project-attempt-integration-state', 'attempt.integration-bind': 'project-attempt-integration-bind', @@ -28,12 +31,12 @@ const operations = Object.freeze({ 'execution.boundary': 'project-execution-boundary', 'pools': 'project-task-pools', 'effect.retry': 'project-effect-retry', 'comment.publish': 'project-github-comment', }) -const queries = new Set(['consultation.get', 'consultation.list', 'handoff.get', 'handoff.list', 'tasks', 'task', 'task.detail', 'inputs', 'sources', 'execution.boundary', 'effects', 'attempt.context', 'attempt.integration-state', 'attempt.log', 'chief.state', 'pools']) +const queries = new Set(['roles', 'role.proposals', 'consultation.get', 'consultation.list', 'handoff.get', 'handoff.list', 'task.history', 'project.snapshot', 'events', 'command.receipt', 'command.result', 'tasks', 'task', 'task.detail', 'inputs', 'sources', 'execution.boundary', 'effects', 'attempt.context', 'attempt.integration-state', 'attempt.log', 'chief.state', 'pools']) const legacy = new Set(Object.values(operations)) const readEndpoints = new Set([...queries].map(name => operations[name]).filter(Boolean)) export const developmentQuery = body => record(body) && body.version === DEVELOPMENT_PROTOCOL && body.kind === 'query' && queries.has(body.name) -export function taskView(state, task, { activity = {} } = {}) { +export function taskView(state, task, { activity = {}, roleCatalog, isIntentCommitted = () => false } = {}) { const c = task.collaboration const candidate = ownEntry(state.projectAttempts ?? {}, task.deliveryAttemptId) const attempt = sameTask(candidate, task) ? candidate : null @@ -41,15 +44,21 @@ export function taskView(state, task, { activity = {} } = {}) { const claim = c?.claim const worker = ownEntry(state.projectHarnessWorkers ?? {}, claim?.workerId) const sdk = sameTask(worker, task) || worker && attempt && worker.projectId === task.projectId && worker.attemptId === attempt.id ? worker : null + const nativeCandidate = ownEntry(state.projectNativeWorkers ?? {}, claim?.workerId) + const nativeWorker = sameTask(nativeCandidate, task) ? nativeCandidate : null + const executionView = executionMetadataView(state, sdk ?? nativeWorker) + const nativeActivated = nativeWorker && isIntentCommitted(nativeWorker.activationEffect) const native = ownEntry(state.bots ?? {}, claim?.workerId) const nativeActive = native?.projectId === task.projectId && (!attempt || native.attemptId === attempt.id) && activity[claim?.workerId] const execution = attempt?.runId ? ownEntry(state.projectExecutions ?? {}, attempt.runId) : null const run = sameTask(execution, task) ? execution : null - const currentRole = (state.projectAgentDefinitions?.[task.projectId]?.roles ?? []).find(row => row.id === c?.botId)?.role ?? (c?.botId?.startsWith(`${task.projectId}:`) && roles.has(c.botId.slice(task.projectId.length + 1)) ? c.botId.slice(task.projectId.length + 1) : null) + const catalog = roleCatalog ?? state.projectAgentDefinitions?.[task.projectId] + const currentRole = (catalog?.roles ?? []).find(row => row.id === c?.botId)?.role ?? claim?.role ?? Object.values(state.projectRetiredRoles ?? {}).find(row => row.projectId === task.projectId && row.id === c?.botId)?.role ?? (c?.botId?.startsWith(`${task.projectId}:`) && roles.has(c.botId.slice(task.projectId.length + 1)) ? c.botId.slice(task.projectId.length + 1) : null) let phase = 'published', waitingFor = null, reason = null if (task.status === 'accepted') phase = 'accepted' else if (task.status === 'cancelled' || c?.status === 'cancelled' || attempt?.status === 'cancelled') phase = 'cancelled' else if (attempt?.cancelRequested || c?.status === 'stopping') { phase = 'stopping'; waitingFor = 'worker-exit' } + else if (nativeWorker?.status === 'stopping' || nativeWorker?.status === 'stop-unconfirmed') { phase = nativeWorker.status === 'stopping' ? 'stopping' : 'blocked'; waitingFor = 'worker-exit'; reason = c?.block?.reason ?? null } else if (task.chiefQuestion) { phase = 'awaiting-owner'; waitingFor = task.chiefQuestion.recipient?.kind === 'personal-agent' ? 'personal-agent' : 'owner'; reason = task.chiefQuestion.reason } else if (task.projectTask?.admission.status !== 'accepted' && task.projectTask) { phase = 'awaiting-chief'; waitingFor = 'admission' } else if (c?.status === 'waiting-children') { phase = 'blocked'; waitingFor = 'children'; reason = c.block?.reason ?? null } @@ -57,9 +66,10 @@ export function taskView(state, task, { activity = {} } = {}) { else if (c?.status === 'blocked' || queue?.status === 'blocked') { phase = 'blocked'; reason = c?.block?.reason ?? queue?.reason ?? null } else if (queue) { phase = queue.status === 'starting' ? 'starting' : 'queued'; waitingFor = queue.reason ? 'resource' : 'capacity'; reason = queue.reason ?? null } else if (c?.status === 'queued') { phase = 'queued'; waitingFor = 'role' } + else if (c?.status === 'claimed' && executionView?.exitConfirmed) { phase = 'stopping'; waitingFor = 'handoff' } else if (c?.status === 'claimed') { if (sdk?.status === 'waiting-input') { phase = 'awaiting-input'; waitingFor = 'owner' } - else if (claim?.status === 'active' && (sdk?.status === 'running' || !sdk && nativeActive)) phase = 'running' + else if (claim?.status === 'active' && (sdk?.status === 'running' || !sdk && nativeActive && (!nativeWorker || nativeActivated))) phase = 'running' else { phase = 'starting'; waitingFor = 'execution' } } else if (task.projectTask?.block && task.status === 'blocked') { phase = 'blocked'; waitingFor = 'chief'; reason = task.projectTask.block.reason } @@ -75,11 +85,11 @@ export function taskView(state, task, { activity = {} } = {}) { } const tests = attempt?.evidence?.tests, review = attempt?.evidence?.review const progress = inspectionFacts(state, { ...task, collaboration: c ? { ...c, history: c.history ?? [] } : undefined, ...(attempt ? {} : { deliveryAttemptId: null }) }, { maxHandoffsWithoutProgress: Infinity, longExecutionMs: Infinity, missingHeartbeatMs: Infinity, longWaitMs: Infinity }) - return { version: DEVELOPMENT_PROTOCOL, projectId: task.projectId, taskId: task.id, publication: 'published', phase, waitingFor, reason, outcome: taskOutcome(state, task), - botId: c?.botId ?? null, currentRole, generation: c?.generation ?? null, attemptId: attempt?.id ?? null, - contract: task.projectTask ? structuredClone(task.projectTask) : null, + return { version: DEVELOPMENT_PROTOCOL, projectId: task.projectId, taskId: task.id, title: typeof task.title === 'string' ? task.title.slice(0, 1000) : '', source: taskProvenance(task), publication: 'published', phase, waitingFor, reason, outcome: taskOutcome(state, task), + botId: c?.botId ?? null, currentRole, roleDefinitionRevision: catalog?.revision ?? 1, execution: executionView, generation: c?.generation ?? null, attemptId: attempt?.id ?? null, + contract: taskContractView(task), executionId: claim?.id ?? run?.id ?? null, workerId: claim?.workerId ?? null, queueId: queue?.id ?? null, candidateSha: sha, - executionGeneration: sdk?.generation ?? null, inputRequestId: sdk?.pending?.id ?? null, + executionGeneration: sdk?.generation ?? nativeWorker?.generation ?? null, inputRequestId: sdk?.pending?.id ?? null, lastMeaningfulProgress: progress.lastMeaningfulProgressAt === null ? null : { at: progress.lastMeaningfulProgressAt, evidence: progress.progressEvidence }, evidence: { tests: tests ? { sha: tests.sha, sha256: tests.sha256, passed: tests.passed === true, current: Boolean(sha && tests.sha === sha) } : null, review: review ? { sha: review.sha, verdict: review.verdict, current: Boolean(sha && review.sha === sha) } : null }, @@ -88,8 +98,13 @@ export function taskView(state, task, { activity = {} } = {}) { } export function createDevelopmentService(rt) { + const receipts = rt.projectEffects ? createCommandReceipts(rt) : null + const management = new Set(['role.decide', 'attempt.stop', 'attempt.cleanup', 'queue.retry', 'chief.inspect', 'chief.configure', 'schedule.configure', 'effect.retry', 'attempt.integration-bind', 'attempt.publication-retry', 'source.subscribe', 'source.disable', 'source.poll']) + const readCommand = new Set(['agent.mode', 'consultation.ask', 'consultation.reply']) + const actionFor = name => ['task.propose', 'proposal.publish', 'comment.publish'].includes(name) ? 'create' : management.has(name) ? 'configure' : readCommand.has(name) ? 'read' : name === 'task.accept' || name === 'task.reject' ? 'accept' : 'task-write' + function access(actor) { - if (!rt.sharedProjectMode?.() || !actor?.memberId || !ownEntry(rt.state.members ?? {}, actor.memberId)) throw failure('Sign in to the Project service', 'bot/unauthorized') + if (!rt.sharedProjectMode?.() || !actor?.memberId || !ownEntry(rt.state.members ?? {}, actor.memberId) || ownEntry(rt.state.members ?? {}, actor.memberId)?.status === 'disabled') throw failure('Sign in to the Project service', 'bot/unauthorized') } function checkedTask(id, projectId, actor) { const task = ownEntry(rt.state.tasks, String(id ?? '')) @@ -99,7 +114,7 @@ export function createDevelopmentService(rt) { } function view(task, actor) { rt.requireProjectAccess(actor, task.projectId) - const result = taskView(rt.state, task, { activity: rt.activityView?.() ?? {} }) + const result = taskView(rt.state, task, { activity: rt.activityView?.() ?? {}, roleCatalog: rt.projectRoleGovernance?.visibleSet(task.projectId) ?? ownEntry(rt.state.projectAgentDefinitions ?? {}, task.projectId), isIntentCommitted: token => Boolean(rt.projectEffects?.intentCommitted(token)) }) const allowed = action => rt.projectCan(actor, task.projectId, action, task) if (result.phase === 'published' || result.phase === 'failed') { if (allowed('create') && allowed('task-write')) result.nextActions.push('task.run') } if (result.queueId && ['queued', 'blocked'].includes(result.phase) && allowed('task-write')) result.nextActions.push('queue.cancel') @@ -113,16 +128,20 @@ export function createDevelopmentService(rt) { return result } function attemptView(attempt) { - const policy = row => row ? { version: row.version, role: row.role, commit: row.commit, profile: row.profile, sha256: row.sha256 } : null - const session = row => { - const { context, policy: rules, ...rest } = row - return { ...rest, policy: policy(rules), context: context ? { version: context.version, sha256: context.sha256, archiveSha256: context.archiveSha256, ...(context.budget ? { budget: context.budget, usage: context.usage } : {}) } : null } + const policy = row => row ? { version: row.version, role: row.role, policyRole: row.policyRole ?? row.role, commit: row.commit, profile: row.profile, sha256: row.sha256 } : null + const session = (row, section = 'roleSessions') => { + const { context, policy: rules } = row + const keys = ['id', 'projectId', 'taskId', 'attemptId', 'role', 'policyRole', 'logicalSessionId', 'definitionRef', 'definitionRevision', 'definitionKey', 'generation', 'poolClaimId', 'status', 'botId', 'sessionId', 'backendSessionId'] + return { ...Object.fromEntries(keys.filter(key => row[key] !== undefined).map(key => [key, structuredClone(row[key])])), + execution: row.execution ? { harness: row.execution.harness, model: row.execution.model ?? null, resourceIds: (row.execution.resourceIds ?? []).slice(0, 20) } : null, + definitionSource: { query: 'attempt.context', attemptId: attempt.id, section }, + policy: policy(rules), context: context ? { version: context.version, sha256: context.sha256, archiveSha256: context.archiveSha256, ...(context.budget ? { budget: context.budget, usage: context.usage } : {}) } : null } } - const { roleSessions, roleSessionHistory, rolePolicies, ...rest } = attempt - return structuredClone({ ...rest, - ...(roleSessions ? { roleSessions: Object.fromEntries(Object.entries(roleSessions).map(([key, row]) => [key, session(row)])) } : {}), - ...(roleSessionHistory ? { roleSessionHistory: roleSessionHistory.map(session) } : {}), - ...(rolePolicies ? { rolePolicies: Object.fromEntries(Object.entries(rolePolicies).map(([key, row]) => [key, policy(row)])) } : {}), + const { roleSessions, roleSessionHistory, rolePolicies } = attempt + return structuredClone({ ...attemptMetadataView(attempt), + ...(roleSessions ? { roleSessions: Object.fromEntries(Object.entries(roleSessions).slice(0, 32).map(([key, row]) => [key, session(row)])) } : {}), + ...(roleSessionHistory ? { roleSessionHistory: roleSessionHistory.slice(-3).map(row => session(row, 'roleSessionHistory')), roleSessionHistoryPage: { total: roleSessionHistory.length, shown: Math.min(3, roleSessionHistory.length), source: { query: 'attempt.context', attemptId: attempt.id, section: 'roleSessionHistory' } } } : {}), + ...(rolePolicies ? { rolePolicies: Object.fromEntries(Object.entries(rolePolicies).slice(0, 32).map(([key, row]) => [key, policy(row)])) } : {}), contextRef: { query: 'attempt.context', attemptId: attempt.id, sections: ['roleSessions', 'roleSessionHistory', 'rolePolicies'] } }) } function detail(body, actor) { @@ -132,11 +151,11 @@ export function createDevelopmentService(rt) { const end = body.beforeAttemptId === undefined ? rows.length : rows.findIndex(row => row.id === body.beforeAttemptId) if (end < 0) throw failure('Attempt cursor is unavailable') const page = rows.slice(Math.max(0, end - limit), end) - return { task: structuredClone(task), attempts: page.map(attemptView), view: view(task, actor), + return { task: taskDetailView(task), attempts: page.map(attemptView), view: view(task, actor), attemptsPage: { total: rows.length, limit, before: end > limit ? page[0].id : null } } } function context(body) { - if (!['roleSessions', 'roleSessionHistory', 'rolePolicies'].includes(body.section)) throw failure('Unknown Attempt context section') + if (!['roleSessions', 'roleSessionHistory', 'rolePolicies', 'evidence', 'feedback', 'requirement', 'failure', 'agentDefinitions', 'integrationAdmissions'].includes(body.section)) throw failure('Unknown Attempt context section') const attempt = ownEntry(rt.state.projectAttempts ?? {}, body.attemptId) const text = JSON.stringify(attempt[body.section] ?? null), offset = body.offset ?? 0, limit = body.limit ?? 8000 if (!Number.isSafeInteger(offset) || offset < 0 || offset > text.length || !Number.isSafeInteger(limit) || limit < 1 || limit > 16000) throw failure('Invalid context page') @@ -153,6 +172,11 @@ export function createDevelopmentService(rt) { return { run: await rt.projectChiefControl.request(body.projectId, 'inspection', { requestId: body.requestId, requestedBy: actor.memberId, ...(task ? { blockedTaskId: task.id } : {}) }) } } async function invoke(endpoint, body, actor) { + if (endpoint === 'task-create') { + let botId = body.botId + if (!botId) botId = (await rt.projectAgents.ensurePublic(body.projectId, actor)).botId + return rt.taskEndpoint(endpoint, { ...body, botId }, actor) + } if (endpoint === 'project-execution-boundary') return rt.projectExecutionBoundary(body, actor) if (endpoint === 'project-sources') return rt.projectSources.query(body, actor) if (endpoint === 'project-source-subscribe') return rt.projectSources.subscribe(body, actor) @@ -172,6 +196,7 @@ export function createDevelopmentService(rt) { if (endpoint === 'project-scheduled-task') return rt.projectScheduledTasks.endpoint(body, actor) if (endpoint === 'project-agent-mode') return rt.projectDelegations.mode(body, actor) if (endpoint === 'project-execution-input') return rt.projectHarness.input(body, actor) + if (['project-roles', 'project-role-proposals', 'project-role-decision'].includes(endpoint)) return rt.projectAgents.endpoint(endpoint, body, actor) if (endpoint === 'project-agent-proposal-apply') return rt.projectAgents.endpoint(endpoint, body, actor) if (endpoint === 'project-task-detail') return detail(body, actor) if (['project-consultation-reply', 'project-consultation-ask', 'project-consultation-get', 'project-consultation-list'].includes(endpoint)) return rt.projectAgents.consultation(endpoint, body, actor) @@ -183,18 +208,25 @@ export function createDevelopmentService(rt) { } return rt.projectDelivery.endpoint(endpoint, body, actor) } + async function commandResult(name, args, actor) { + const result = await invoke(operations[name], args, actor) + if (name === 'task.propose') return taskDetailView(result) + if (!record(result)) return result + return { ...result, ...(result.attempt ? { attempt: attemptView(result.attempt) } : {}), ...(result.task ? { task: taskDetailView(result.task) } : {}), + ...(result.run ? { run: Object.fromEntries(['id', 'projectId', 'taskId', 'kind', 'status', 'startedAt', 'completedAt', 'createdAt'].filter(key => result.run[key] !== undefined).map(key => [key, result.run[key]])), runRef: { query: 'chief.state', projectId: args.projectId } } : {}) } + } async function envelope(body, actor) { if (!record(body) || body.version !== DEVELOPMENT_PROTOCOL || !['command', 'query'].includes(body.kind) || !record(body.payload ?? {})) throw failure('Unsupported Project development request') const { name, projectId } = body rt.requireProjectAccess(actor, projectId) - if (!Object.hasOwn(operations, name) && !['tasks', 'task', 'attempt.context', 'effects'].includes(name)) throw failure('Unknown Project development operation') + if (!Object.hasOwn(operations, name) && !['tasks', 'task', 'task.history', 'project.snapshot', 'events', 'command.receipt', 'command.result', 'attempt.context', 'effects'].includes(name)) throw failure('Unknown Project development operation') if (queries.has(name) !== (body.kind === 'query')) throw failure('Operation kind does not match the contract') if (body.kind === 'command' && !stableId(body.requestId)) throw failure('A command needs a stable requestId') const args = { ...(body.payload ?? {}), projectId, ...(body.requestId !== undefined ? { requestId: body.requestId } : {}) } if (body.payload?.projectId !== undefined && body.payload.projectId !== projectId) throw failure('Payload belongs to another Project', 'bot/forbidden') // These targets have their own authority in the existing services, but the // envelope must also reject a target in another Project the actor can access. - for (const [key, table] of [['proposalId', rt.state.projectAgentProposals], ['workerId', rt.state.projectHarnessWorkers]]) { + for (const [key, table] of [['proposalId', ['role.decide', 'role.proposals'].includes(name) ? rt.state.projectRoleProposals : rt.state.projectAgentProposals], ['workerId', rt.state.projectHarnessWorkers]]) { if (args[key] === undefined) continue const target = ownEntry(table ?? {}, String(args[key])) if (!target || target.projectId !== projectId) throw failure('Target is unavailable in this Project', 'bot/forbidden') @@ -210,10 +242,26 @@ export function createDevelopmentService(rt) { if (task && selected.id !== task.id || args.taskId && selected.id !== args.taskId) throw failure('Targets belong to different Tasks', 'bot/forbidden') task = selected } - if (body.expectedGeneration !== undefined && (!Number.isSafeInteger(body.expectedGeneration) || body.expectedGeneration < 0 || !task || task.collaboration?.generation !== body.expectedGeneration)) throw failure('Task generation changed', 'bot/conflict') - if (body.expectedCandidateSha !== undefined && (body.expectedCandidateSha !== null && (typeof body.expectedCandidateSha !== 'string' || !/^[a-f0-9]{40}$/.test(body.expectedCandidateSha)) || !task || taskView(rt.state, task).candidateSha !== body.expectedCandidateSha)) throw failure('Task candidate changed', 'bot/conflict') + if (!task && name === 'execution.input') { + const worker = ownEntry(rt.state.projectHarnessWorkers ?? {}, args.workerId) + if (worker) task = checkedTask(worker.taskId ?? ownEntry(rt.state.projectAttempts ?? {}, worker.attemptId)?.taskId, projectId, actor) + } + const check = () => { + if (task) task = checkedTask(task.id, projectId, actor) + if (body.expectedGeneration !== undefined && (!Number.isSafeInteger(body.expectedGeneration) || body.expectedGeneration < 0 || !task || task.collaboration?.generation !== body.expectedGeneration)) throw failure('Task generation changed', 'bot/conflict') + if (body.expectedCandidateSha !== undefined && (body.expectedCandidateSha !== null && (typeof body.expectedCandidateSha !== 'string' || !/^[a-f0-9]{40}$/.test(body.expectedCandidateSha)) || !task || taskView(rt.state, task).candidateSha !== body.expectedCandidateSha)) throw failure('Task candidate changed', 'bot/conflict') + } + if (body.kind === 'query') check() let result - if (name === 'effects') result = { effects: rt.projectEffects.list(projectId, actor, args) } + if (name === 'command.receipt') { if (!receipts) throw failure('Command receipts are unavailable'); result = receipts.get(projectId, args.commandRequestId, actor, args.commandName) } + else if (name === 'command.result') { if (!receipts) throw failure('Command receipts are unavailable'); result = receipts.result(projectId, args, actor) } + else if (name === 'events') result = rt.committedProjectEvents.query(projectId, args) + else if (name === 'project.snapshot') result = rt.committedProjectEvents.snapshot(projectId, args) + else if (name === 'effects') result = { effects: rt.projectEffects.list(projectId, actor, args) } + else if (name === 'task.history') { + const source = checkedTask(args.taskId, projectId, actor) + result = { taskId: source.id, section: args.section, ...sourcePage(taskHistorySource(source, args.section), args, text => createHash('sha256').update(text).digest('hex')) } + } else if (name === 'task') result = { task: view(checkedTask(args.taskId, projectId, actor), actor) } else if (name === 'attempt.context') { if (!args.attemptId) throw failure('Attempt context needs an Attempt') @@ -227,10 +275,20 @@ export function createDevelopmentService(rt) { if (args.after !== undefined && index < 0) throw failure('Task cursor is unavailable') const page = rows.slice(index + 1, index + 1 + limit) result = { tasks: page.map(row => view(row, actor)), next: index + 1 + page.length < rows.length ? page.at(-1).id : null } - } else result = await invoke(operations[name], args, actor) + } else if (body.kind === 'command' && receipts) { + const delivered = await receipts.run({ projectId, name, requestId: body.requestId, args, actor, action: actionFor(name), taskId: task?.id, + expectedGeneration: body.expectedGeneration, expectedCandidateSha: body.expectedCandidateSha, + reconcile: name === 'task.run' ? async () => ({ result: await commandResult(name, args, actor) }) : ['task.propose', 'proposal.publish'].includes(name) ? async row => { + const existing = Object.values(rt.state.tasks).find(task => task.projectId === projectId && task.commandSource?.commandReceiptId === row.id) + const proof = existing && Object.values(rt.state.projectEffects ?? {}).find(effect => effect.kind === 'task.intake' && effect.taskId === existing.id && rt.projectEffects.intentCommitted({ id: effect.id, generation: effect.generation })) + return proof ? { result: taskDetailView(existing) } : null + } : undefined, + }, () => commandResult(name, args, actor), check) + return { version: DEVELOPMENT_PROTOCOL, requestId: body.requestId, ...delivered } + } else { check(); result = await invoke(operations[name], args, actor) } return { version: DEVELOPMENT_PROTOCOL, ...(body.requestId !== undefined ? { requestId: body.requestId } : {}), result } } - return { view: (task, actor) => { access(actor); return view(task, actor) }, + return { commandSource: projectId => receipts?.source(projectId) ?? null, view: (task, actor) => { access(actor); return view(task, actor) }, isQuery: developmentQuery, // Trusted host ports. Public HTTP never accepts role tokens or model results. roleResult: row => rt.projectPools.result(row), diff --git a/core/src/host/project-effects.js b/core/src/host/project-effects.js index 5a13aece..96b0b996 100644 --- a/core/src/host/project-effects.js +++ b/core/src/host/project-effects.js @@ -1,5 +1,6 @@ // Persist intent with Project state, then execute through registered host adapters. import { createHash } from 'node:crypto' +import { commandEffectVisible } from './project-command-receipts.js' import { ownEntry, setEntry } from './store.js' export const EFFECT_PROTOCOL = 'project-effect/v1' @@ -177,13 +178,14 @@ export function createProjectEffects(rt, { now = Date.now } = {}) { function list(projectId, actor, { taskId, limit = 50 } = {}) { rt.requireProjectAccess(actor, projectId) if (!Number.isSafeInteger(limit) || limit < 1 || limit > 200) throw fail('Effect page limit must be between 1 and 200') - return Object.values(rt.state.projectEffects ?? {}).filter(row => row.projectId === projectId && (!taskId || row.taskId === taskId)).sort((a, b) => b.updatedAt - a.updatedAt || a.id.localeCompare(b.id)).slice(0, limit) + return Object.values(rt.state.projectEffects ?? {}).filter(row => row.projectId === projectId && commandEffectVisible(rt.state, row, actor) && (!taskId || row.taskId === taskId)).sort((a, b) => b.updatedAt - a.updatedAt || a.id.localeCompare(b.id)).slice(0, limit) .map(({ payload: _payload, history: _history, key: _key, ...row }) => ({ ...copy(row), committed: durable.get(row.id) === stamp(row) })) } async function retry(body, actor) { const row = ownEntry(rt.state.projectEffects ?? {}, body.effectId) if (!row || row.projectId !== body.projectId) throw fail('Effect is unavailable in this Project') rt.requireProjectAccess(actor, row.projectId, 'configure') + if (!commandEffectVisible(rt.state, row, actor)) throw fail('Effect is private to another command principal') if (rt.readOnly?.() !== undefined || !/^[A-Za-z0-9_-]{1,100}$/.test(body.requestId ?? '')) throw fail('Effect retry needs a writable host and stable requestId') if (row.version !== EFFECT_PROTOCOL) throw fail('Effect protocol upgrade is required') const prior = row.retries?.find(item => item.requestId === body.requestId) diff --git a/core/src/host/project-pools.js b/core/src/host/project-pools.js index 7db6d0d9..26dc2117 100644 --- a/core/src/host/project-pools.js +++ b/core/src/host/project-pools.js @@ -55,9 +55,16 @@ export function handoffContract(state, task, receipt) { } export function createProjectPools(rt) { - let adapter, tail = Promise.resolve() + let adapter, acceptanceEffects, tail = Promise.resolve() let recovered = false const serial = fn => { const result = tail.then(fn, fn); tail = result.catch(() => {}); return result } + function acceptanceAdapter() { + if (rt.projectEffects && acceptanceEffects !== rt.projectEffects) { + rt.projectEffects.register('pool.accept', { run: effect => ({ claimId: effect.payload.claimId, handoffId: effect.payload.handoffId, status: 'durable-acceptance' }) }, { replace: true }) + acceptanceEffects = rt.projectEffects + } + } + acceptanceAdapter() // Register on restart even when no new claim has been made. const taskOf = id => { const task = ownEntry(rt.state.tasks, id); if (!task?.projectId) throw refuse('Unknown Project Task'); return task } const definitions = projectId => rt.projectAgents.deliveryDefinitions(projectId) const targetOf = (projectId, botId) => { @@ -86,6 +93,7 @@ export function createProjectPools(rt) { async function current(task, token) { const claim = task.collaboration?.claim if (!claim || claim.id !== token?.id || claim.generation !== token?.generation || claim.botId !== token?.botId || claim.status !== 'active') throw refuse('Claim is stale or belongs to another Bot') + if (claim.acceptanceEffect && !rt.projectEffects?.intentCommitted(claim.acceptanceEffect)) throw refuse('Claim acceptance has no confirmed durable intent') const requirementsRevision = claim.requirementsRevision ?? rt.state.projectAttempts?.[claim.attemptId]?.requirement?.projectTaskRevision const currentRequirements = () => !task.projectTask || requirementsRevision === undefined || requirementsRevision === task.projectTask.requirements.revision if (!currentRequirements()) throw refuse('Task requirements version changed before execution decision') @@ -198,6 +206,7 @@ export function createProjectPools(rt) { await authority(task) const prior = c?.history.find(row => row.acceptance?.requestId === requestId) if (prior) { + if (prior.acceptance.effect && !rt.projectEffects?.intentCommitted(prior.acceptance.effect)) throw refuse('Handoff acceptance is not durably committed') if (prior.id !== handoffId || prior.to !== botId) throw refuse('Accept idempotency key was reused for another handoff') const execution = prior.execution ?? (c.claim?.id === prior.claimId ? c.claim : null) return execution ? clone(execution) : { ...clone(prior.acceptance), id: prior.claimId, status: 'historical' } @@ -231,11 +240,33 @@ export function createProjectPools(rt) { const limit = rt.config.projectHandoff?.quota?.steps if (Number.isFinite(limit) && (c.usage?.steps ?? 0) >= limit) { c.status = 'blocked'; c.block = { reason: 'Handoff refused (quota): execution step quota is exhausted', at: Date.now() }; await rt.save(); throw refuse(c.block.reason) } const claim = { id: `claim-${randomUUID()}`, botId, role: definition.role, policyRole: definition.policyRole ?? definition.role, ...(task.projectTask ? { requirementsRevision: task.projectTask.requirements.revision } : {}), generation: ++c.generation, taskId: task.id, projectId: task.projectId, attemptId: c.attemptId, status: 'active', createdAt: Date.now() } + const before = { generation: expectedGeneration, status: c.status, claim: c.claim, usage: c.usage === undefined ? undefined : clone(c.usage), receipt: clone(receipt), wait: task.projectTask?.wait ? clone(task.projectTask.wait) : null } c.usage ??= {}; c.usage.steps = (c.usage.steps ?? 0) + 1 if (receipt.purpose === 'children-completed' && task.projectTask?.wait?.status === 'queued') Object.assign(task.projectTask.wait, { status: 'resumed', resumedClaimId: claim.id, resumedAt: Date.now() }) c.claim = claim; c.status = 'claimed'; receipt.status = 'accepted'; receipt.acceptedAt = Date.now(); receipt.claimId = claim.id receipt.acceptance = { claimId: claim.id, botId: claim.botId, generation: claim.generation, acceptedAt: receipt.acceptedAt, ...(requestId ? { requestId } : {}), ...(claim.chiefRunId ? { chiefRunId: claim.chiefRunId } : {}) } - await rt.save() + let intent + try { + if (rt.projectEffects) { + acceptanceAdapter() + intent = rt.projectEffects.stage({ kind: 'pool.accept', key: claim.id, projectId: task.projectId, taskId: task.id, + payload: { claimId: claim.id, handoffId: receipt.id, attemptId: claim.attemptId, botId: claim.botId, generation: claim.generation } }) + claim.acceptanceEffect = intent + receipt.acceptance.effect = intent + await rt.projectEffects.persist(intent) + } else await rt.save() // In-memory/legacy hosts without the durable Effects service. + } catch (error) { + const uncommitted = !intent || rt.projectEffects.discardUncommitted(intent) + if (uncommitted && c.claim === claim && c.generation === claim.generation && c.currentHandoffId === receipt.id) { + c.generation = before.generation; c.status = before.status + if (before.claim === undefined) delete c.claim; else c.claim = before.claim + if (before.usage === undefined) delete c.usage; else c.usage = before.usage + for (const key of Object.keys(receipt)) delete receipt[key] + Object.assign(receipt, before.receipt) + if (before.wait) task.projectTask.wait = before.wait + } + throw error + } return clone(claim) } async function decide(taskId, token, decision) { diff --git a/core/src/host/project-role-governance.js b/core/src/host/project-role-governance.js index 1757d0d2..a1077a29 100644 --- a/core/src/host/project-role-governance.js +++ b/core/src/host/project-role-governance.js @@ -16,7 +16,7 @@ export function createProjectRoleGovernance(rt, ports) { const previousSets = new Map() rt.projectEffects?.register('role.governance', { run: effect => ({ proposalId: effect.payload.proposalId ?? null, changeId: effect.payload.changeId ?? null, status: 'durable-role-record' }) }) const committed = token => Boolean(token && rt.projectEffects?.intentCommitted(token)) - function visibleSet(projectId, current = ports.definitionSet(projectId)) { + function visibleSet(projectId, current = ports.readDefinitionSet ? ports.readDefinitionSet(projectId) : ports.definitionSet(projectId)) { if (!current.changeEffect || committed(current.changeEffect)) return current const previous = previousSets.get(projectId) if (!previous) throw fail('Role definition change is not committed', 'bot/conflict') diff --git a/core/src/host/project-service-events.js b/core/src/host/project-service-events.js new file mode 100644 index 00000000..2e455d55 --- /dev/null +++ b/core/src/host/project-service-events.js @@ -0,0 +1,136 @@ +import { executionMetadataView } from './project-service-views.js' +// A committed read model over the original Store. No event reception grants +// initiative, and private conversations/credentials never enter this projection. +import { createHash } from 'node:crypto' +const hash = value => createHash('sha256').update(JSON.stringify(value)).digest('hex') +const copy = value => structuredClone(value) +const own = (table, id) => table && Object.hasOwn(table, id) ? table[id] : undefined +const text = value => typeof value === 'string' ? value.slice(0, 1000) : '' +const fail = (message, code = 'bot/invalid') => Object.assign(new Error(message), { code }) +const VERSION = 'project-event/v1' +function projections(state) { + const rows = new Map() + for (const row of Object.values(state.projects ?? {})) { + if (!row?.id) continue + const data = { id: row.id, name: text(row.name), status: row.status, goal: text(row.goal), scope: text(row.scope), repository: row.repository ?? null, roleDefinitionRevision: own(state.projectAgentDefinitions, row.id)?.revision ?? 1 } + rows.set(`project:${row.id}`, { projectId: row.id, entityType: 'project', id: row.id, data, fingerprint: hash([data, row.roles ?? {}, row.repositories ?? []]) }) + } + for (const row of Object.values(state.tasks ?? {})) { + if (!row?.projectId || !own(state.projects, row.projectId)) continue + const c = row.collaboration, data = { id: row.id, projectId: row.projectId, title: text(row.title), status: row.status, + generation: c?.generation ?? null, collaborationStatus: c?.status ?? null, claimId: c?.claim?.id ?? null, workerId: c?.claim?.workerId ?? null, botId: c?.botId ?? null, attemptId: row.deliveryAttemptId ?? null, + admission: row.projectTask?.admission.status ?? null, requirementsRevision: row.projectTask?.requirements.revision ?? null, + waitStatus: row.projectTask?.wait?.status ?? null, block: Boolean(c?.block || row.projectTask?.block), resultDecisionId: row.completedBy?.decisionId ?? null } + rows.set(`task:${row.id}`, { projectId: row.projectId, entityType: 'task', id: row.id, data, fingerprint: hash([data, row.note ?? '', row.criteria ?? '', row.projectTask?.requirements ?? null]) }) + } + for (const row of Object.values(state.projectAttempts ?? {})) { + const task = own(state.tasks, row.taskId) + if (!task || task.projectId !== row.projectId || !own(state.projects, row.projectId)) continue + const data = { id: row.id, taskId: row.taskId, projectId: row.projectId, status: row.status, candidateSha: row.candidateSha ?? null, + tests: row.evidence?.tests ? { sha: row.evidence.tests.sha, passed: row.evidence.tests.passed === true } : null, + review: row.evidence?.review ? { sha: row.evidence.review.sha, verdict: row.evidence.review.verdict } : null, + publication: row.pr ? { number: row.pr.number, sha: row.pr.sha } : null, stopping: Boolean(row.cancelRequested) } + rows.set(`attempt:${row.id}`, { projectId: row.projectId, entityType: 'attempt', id: row.id, data, fingerprint: hash(data) }) + } + for (const row of Object.values(state.projectRoleProposals ?? {})) { + if (!row?.projectId || !own(state.projects, row.projectId)) continue + const data = { id: row.id, projectId: row.projectId, role: row.role, action: row.action, status: row.status, baseRevision: row.baseRevision, createdAt: row.createdAt ?? null, decisionAt: row.decision?.at ?? null } + rows.set(`role-proposal:${row.id}`, { projectId: row.projectId, entityType: 'role-proposal', id: row.id, data, fingerprint: hash(data) }) + } + for (const [namespace, entityType] of [['projectHarnessWorkers', 'execution'], ['projectNativeWorkers', 'execution'], ['projectResourceLeases', 'resource'], ['projectTaskInputs', 'input']]) { + for (const row of Object.values(state[namespace] ?? {})) { + const attempt = own(state.projectAttempts, row.attemptId) + const task = own(state.tasks, row.taskId ?? attempt?.taskId) + if (!task || task.projectId !== row.projectId || row.chiefRunId || !own(state.projects, row.projectId)) continue + const data = { id: row.id, projectId: row.projectId, taskId: task.id, attemptId: row.attemptId ?? null, status: row.status, + ...(entityType === 'execution' ? executionMetadataView(state, row) : entityType === 'input' ? { kind: row.kind, sha256: row.sha256, deliveries: row.deliveries?.length ?? 0, readConfirmed: row.deliveries?.some(delivery => Boolean(delivery.readAt)) ?? false } : { role: row.role ?? null, resourceId: row.resourceId ?? null, generation: row.generation ?? null, pending: row.pending ? { id: row.pending.id, kind: row.pending.kind } : null }) } + rows.set(`${entityType}:${row.id}`, { projectId: row.projectId, entityType, id: row.id, data, fingerprint: hash(data) }) + } + } + return rows +} +export function createProjectServiceEvents({ maxPerProject = 1024, maxAgeMs = 24 * 60 * 60 * 1000, now = Date.now } = {}) { + if (!Number.isSafeInteger(maxPerProject) || maxPerProject < 1 || maxPerProject > 4096 || !Number.isSafeInteger(maxAgeMs) || maxAgeMs < 1000) throw fail('Invalid Project event retention policy') + let committed = { projections: new Map(), streams: {}, events: {} } + const epoch = (state, projectId) => hash([VERSION, state.workspaceId ?? 'legacy', projectId, own(state.projects, projectId)?.createdAt ?? 0]).slice(0, 32) + function streamFor(state, projectId, streams) { + const prior = own(streams, projectId) + if (prior) return prior + const stream = { epoch: epoch(state, projectId), nextSequence: 1, floor: 0, revision: 0, lastAt: 0 } + Object.defineProperty(streams, projectId, { value: stream, enumerable: true, writable: true, configurable: true }) + return stream + } + function load(state) { + const streams = copy(state.projectServiceEventStreams ?? {}), events = copy(state.projectServiceEvents ?? {}) + for (const row of Object.values(state.projects ?? {})) if (row?.id) streamFor(state, row.id, streams) + for (const [projectId, stream] of Object.entries(streams)) if (!/^[a-f0-9]{32}$/.test(stream.epoch) || !Number.isSafeInteger(stream.nextSequence) || !Number.isSafeInteger(stream.floor) || stream.floor < 0 || stream.nextSequence <= stream.floor || !Number.isSafeInteger(stream.revision) || stream.revision < 0 || stream.lastAt !== undefined && (!Number.isSafeInteger(stream.lastAt) || stream.lastAt < 0)) throw fail(`Project event stream metadata is malformed: ${projectId}`) + for (const row of Object.values(events)) if (row.version !== VERSION || !own(streams, row.projectId) || row.epoch !== streams[row.projectId].epoch || !Number.isSafeInteger(row.sequence) || row.sequence <= streams[row.projectId].floor || row.sequence >= streams[row.projectId].nextSequence) throw fail('Project event record is outside its committed stream') + for (const [projectId, stream] of Object.entries(streams)) { + const rows = Object.values(events).filter(row => row.projectId === projectId).sort((a, b) => a.sequence - b.sequence) + if (rows.length !== stream.nextSequence - stream.floor - 1 || rows.some((row, index) => row.sequence !== stream.floor + index + 1 || !Number.isSafeInteger(row.at) || row.at < 0 || index && row.at < rows[index - 1].at)) throw fail('Committed Project events have a gap, duplicate or invalid timestamp') + stream.lastAt = Math.max(stream.lastAt ?? 0, rows.at(-1)?.at ?? 0) + } + committed = { projections: projections(state), streams, events } + } + function prepare(snapshot) { + const nextProjections = projections(snapshot), streams = copy(committed.streams), events = copy(committed.events), changes = [], touched = new Set() + for (const [key, row] of nextProjections) if (committed.projections.get(key)?.fingerprint !== row.fingerprint) changes.push({ ...row, operation: 'upsert' }) + for (const [key, row] of committed.projections) if (!nextProjections.has(key)) changes.push({ ...row, operation: 'delete' }) + for (const row of changes) { + const stream = streamFor(snapshot, row.projectId, streams) + if (!touched.has(row.projectId)) { ++stream.revision; touched.add(row.projectId) } + const sequence = stream.nextSequence++, event = { version: VERSION, projectId: row.projectId, epoch: stream.epoch, sequence, + revision: stream.revision, at: Math.max(stream.lastAt ?? 0, now()), type: `${row.entityType}.${row.operation === 'delete' ? 'deleted' : 'changed'}`, entityType: row.entityType, entityId: row.id, + data: row.operation === 'delete' ? { id: row.id } : row.data } + stream.lastAt = event.at + Object.defineProperty(events, hash([row.projectId, stream.epoch, sequence]), { value: event, enumerable: true, writable: true, configurable: true }) + } + for (const [projectId, stream] of Object.entries(streams)) { + const rows = Object.entries(events).filter(([, row]) => row.projectId === projectId).sort((a, b) => a[1].sequence - b[1].sequence) + const retained = rows.filter(([, row]) => row.at >= now() - maxAgeMs).slice(-maxPerProject) + const keep = new Set(retained.map(([key]) => key)) + for (const [key, row] of rows) if (!keep.has(key)) { stream.floor = Math.max(stream.floor, row.sequence); delete events[key] } + } + snapshot.projectServiceEventStreams = streams + snapshot.projectServiceEvents = events + return { snapshot, committed: { projections: nextProjections, streams, events } } + } + function saved(prepared) { committed = prepared.committed } + function cursor(projectId, stream, sequence) { return Buffer.from(JSON.stringify({ v: 1, p: projectId, e: stream.epoch, s: sequence })).toString('base64url') } + function streamOf(projectId) { + const stream = own(committed.streams, projectId) + if (!stream) throw fail('Project has no committed event snapshot', 'bot/not-found') + return stream + } + function query(projectId, { after, limit = 100 } = {}) { + if (!Number.isSafeInteger(limit) || limit < 1 || limit > 100) throw fail('Invalid event page limit') + const stream = streamOf(projectId) + const expired = Object.values(committed.events).filter(row => row.projectId === projectId && row.at < now() - maxAgeMs) + const floor = Math.max(stream.floor, ...expired.map(row => row.sequence)) + let sequence = floor + if (after !== undefined) { + if (typeof after !== 'string' || after.length > 600 || !/^[A-Za-z0-9_-]+$/.test(after)) throw fail('Invalid event cursor') + let parsed + try { parsed = JSON.parse(Buffer.from(after, 'base64url').toString('utf8')) } catch { throw fail('Invalid event cursor') } + if (parsed.v !== 1 || parsed.p !== projectId || !Number.isSafeInteger(parsed.s) || parsed.s < 0) throw fail('Event cursor belongs to another scope', 'bot/conflict') + if (parsed.e !== stream.epoch || parsed.s < floor || parsed.s > stream.nextSequence - 1) throw Object.assign(fail('Event cursor expired; read a current Project snapshot', 'bot/cursor-expired'), { nextAction: 'project.snapshot', details: { nextAction: 'project.snapshot' } }) + sequence = parsed.s + } + const rows = Object.values(committed.events).filter(row => row.projectId === projectId && row.sequence > sequence && row.at >= now() - maxAgeMs).sort((a, b) => a.sequence - b.sequence).slice(0, limit) + return { version: VERSION, projectId, revision: stream.revision, events: rows.map(row => ({ ...copy(row), cursor: cursor(projectId, stream, row.sequence) })), + next: cursor(projectId, stream, rows.at(-1)?.sequence ?? sequence), head: cursor(projectId, stream, stream.nextSequence - 1), oldest: cursor(projectId, stream, floor) } + } + function snapshot(projectId, { after, limit = 50, expectedRevision, entityType = 'task' } = {}) { + const stream = streamOf(projectId) + if (!['task', 'attempt', 'execution', 'resource', 'input', 'role-proposal'].includes(entityType)) throw fail('Unknown public Project snapshot entity type') + if (after !== undefined && expectedRevision === undefined) throw fail('Continuing a Project snapshot requires its expectedRevision', 'bot/conflict') + if (!Number.isSafeInteger(limit) || limit < 1 || limit > 100 || expectedRevision !== undefined && expectedRevision !== stream.revision) throw fail('Project snapshot changed or page limit is invalid', 'bot/conflict') + const rows = [...committed.projections.values()].filter(row => row.projectId === projectId && row.entityType === entityType).sort((a, b) => a.id.localeCompare(b.id)) + const index = after === undefined ? -1 : rows.findIndex(row => row.id === after) + if (after !== undefined && index < 0) throw fail('Project snapshot cursor is unavailable') + const page = rows.slice(index + 1, index + 1 + limit), project = committed.projections.get(`project:${projectId}`) + return { version: VERSION, project: project ? copy(project.data) : null, revision: stream.revision, cursor: cursor(projectId, stream, stream.nextSequence - 1), + ...(entityType === 'task' ? { tasks: page.map(row => copy(row.data)) } : { entityType, entities: page.map(row => copy(row.data)) }), next: index + 1 + page.length < rows.length ? page.at(-1).id : null } + } + return { load, prepare, saved, query, snapshot } +} diff --git a/core/src/host/project-service-sse.js b/core/src/host/project-service-sse.js new file mode 100644 index 00000000..67469aa6 --- /dev/null +++ b/core/src/host/project-service-sse.js @@ -0,0 +1,69 @@ +// Request-scoped transport over authenticated service queries. No background +// project scheduler, bearer-in-URL, raw DSH stream or parallel event authority. +import { setTimeout as delay } from 'node:timers/promises' +import { once } from 'node:events' + +export function createProjectSse({ pull, pollMs = 1000, maxStreams = 64 }) { + if (!Number.isSafeInteger(pollMs) || pollMs < 10 || pollMs > 60000 || !Number.isSafeInteger(maxStreams) || maxStreams < 1 || maxStreams > 256) throw new Error('Invalid Project SSE transport limits') + const active = new Set() + async function handle(req, res, url, respond) { + const token = req.headers['x-flybot-member-session'], projectId = url.searchParams.get('projectId') + const invalid = [...url.searchParams.keys()].some(key => !['projectId', 'after'].includes(key) || url.searchParams.getAll(key).length !== 1) + if (invalid || !/^[A-Za-z0-9_-]{1,100}$/.test(projectId ?? '')) return respond(400, { ok: false, error: { code: 'bot/invalid', message: 'Choose one Project and an optional event cursor' } }) + if (typeof token !== 'string' || !token || token.length > 8192) return respond(401, { ok: false, error: { code: 'bot/unauthorized', message: 'An authenticated member or personal-agent header is required' } }) + let cursor = req.headers['last-event-id'] ?? url.searchParams.get('after') ?? undefined + if (url.searchParams.has('after') && req.headers['last-event-id'] !== undefined && req.headers['last-event-id'] !== url.searchParams.get('after') || cursor !== undefined && (typeof cursor !== 'string' || cursor.length > 600 || !/^[A-Za-z0-9_-]+$/.test(cursor))) return respond(400, { ok: false, error: { code: 'bot/invalid', message: 'Invalid or conflicting event cursor' } }) + if (active.size >= maxStreams) return respond(429, { ok: false, error: { code: 'bot/busy', message: 'Project event transport is at capacity', details: { nextAction: 'retry' } } }) + const controller = new AbortController(), { signal } = controller + active.add(controller) + const close = () => controller.abort() + res.once('close', close) + const frame = (type, data, id) => `${id ? `id: ${id}\n` : ''}event: ${type}\ndata: ${JSON.stringify(data)}\n\n` + function send(type, data, id) { + if (signal.aborted || res.destroyed) return false + return res.write(frame(type, data, id)) + } + try { + const first = await pull({ projectId, name: cursor === undefined ? 'project.snapshot' : 'events', payload: cursor === undefined ? {} : { after: cursor, limit: 100 }, token, signal }) + if (signal.aborted) return + if (cursor === undefined && (first?.project?.id !== projectId || typeof first.cursor !== 'string' || !/^[A-Za-z0-9_-]{1,600}$/.test(first.cursor))) throw new Error('Invalid scoped Project snapshot') + res.writeHead(200, { 'content-type': 'text/event-stream; charset=utf-8', 'cache-control': 'no-store', 'x-accel-buffering': 'no' }) + res.flushHeaders() + let page = first + if (cursor === undefined) { + cursor = first.cursor + if (!send('snapshot', first, cursor)) await once(res, 'drain', { signal }) + page = null + } + while (!signal.aborted) { + page ??= await pull({ projectId, name: 'events', payload: { after: cursor, limit: 100 }, token, signal }) + if (signal.aborted) break + if (!Array.isArray(page.events)) throw new Error('Invalid Project event page') + let backpressured = false + for (const row of page.events) { + if (row.projectId !== projectId || typeof row.cursor !== 'string' || !/^[A-Za-z0-9_-]{1,600}$/.test(row.cursor)) throw new Error('Project event scope changed') + cursor = row.cursor + if (!send('fact', row, cursor)) { + await once(res, 'drain', { signal }) + backpressured = true + break // Reauthenticate/requery instead of draining a stale buffered batch. + } + } + if (!backpressured && !page.events.length && !res.destroyed && !res.write(': keep-alive\n\n')) await once(res, 'drain', { signal }) + page = null + if (!backpressured) await delay(pollMs, undefined, { signal }) + } + } catch (error) { + if (!signal.aborted) { + const code = typeof error.code === 'string' && error.code.startsWith('bot/') ? error.code : 'bot/unavailable' + const details = { nextAction: ['bot/cursor-expired', 'bot/conflict'].includes(code) ? 'project.snapshot' : ['bot/unauthorized', 'bot/forbidden'].includes(code) ? 'authenticate' : 'retry' } + if (!res.headersSent) respond(code === 'bot/unauthorized' ? 401 : code === 'bot/forbidden' ? 403 : ['bot/cursor-expired', 'bot/conflict'].includes(code) ? 409 : code === 'bot/invalid' ? 400 : 503, { ok: false, error: { code, message: 'Project event request could not continue', details } }) + else send(['bot/unauthorized', 'bot/forbidden'].includes(code) ? 'access-revoked' : 'reset-required', { code, details }) + } + } finally { + controller.abort(); active.delete(controller); res.removeListener('close', close) + if (!res.writableEnded && !res.destroyed) res.end() + } + } + return { handle, close() { for (const controller of active) controller.abort() }, active: () => active.size } +} diff --git a/core/src/host/project-service-views.js b/core/src/host/project-service-views.js new file mode 100644 index 00000000..510acf31 --- /dev/null +++ b/core/src/host/project-service-views.js @@ -0,0 +1,93 @@ +import { nativeExitConfirmed } from './project-native-sessions.js' +import { confirmedWorkerExit } from './project-task-sessions.js' +// Public projections retain references to exact scoped sources; never copy an +// entire growing lifecycle or caller authentication context into every list row. +const pick = (row, keys) => Object.fromEntries(keys.filter(key => row?.[key] !== undefined).map(key => [key, structuredClone(row[key])])) +const short = (value, max = 2000) => typeof value === 'string' ? value.slice(0, max) : value +const reference = (taskId, section, count) => ({ query: 'task.history', taskId, section, ...(count === undefined ? {} : { count }) }) +export function taskProvenance(task) { + const source = task.commandSource?.origin + const agentId = source?.agentId ?? task.personalAgentSource?.agentId + return { kind: agentId ? 'personal-agent' : source?.kind ?? 'human', memberId: task.requesterId ?? null, ...(agentId ? { agentId } : {}), ...(source?.botId ? { botId: source.botId } : {}), ...(task.personalAgentSource?.approvedBy ? { approvedBy: task.personalAgentSource.approvedBy } : {}), scope: 'project-published' } +} +export function taskContractView(task) { + const row = task.projectTask + if (!row) return null + return { version: row.version, projectionVersion: 'project-task-contract-summary/v1', admission: { ...pick(row.admission, ['status', 'revision']), latestDecision: row.admission.decisions?.at(-1) ? pick(row.admission.decisions.at(-1), ['id', 'runId', 'at']) : null }, + requirements: { ...pick(row.requirements, ['revision', 'updatedAt']), ...Object.fromEntries(['title', 'criteria', 'impact', 'scope'].filter(key => row.requirements[key] !== undefined).map(key => [key, short(row.requirements[key])])) }, + parentTaskId: row.parentTaskId ?? null, + wait: row.wait ? { ...pick(row.wait, ['id', 'status', 'botId', 'claimId', 'resumedClaimId', 'at', 'resumedAt']), resumePoint: short(row.wait.resumePoint) } : null, + dependencies: (row.dependencies ?? []).slice(0, 16).map(dependency => ({ taskId: dependency.taskId, status: dependency.status, + result: dependency.result ? pick(dependency.result, ['kind', 'taskId', 'sourceTaskId', 'replacementTaskId', 'replacementVersion', 'decisionId', 'attemptId', 'candidateSha', 'requirementsRevision', 'integrated']) : null })), + block: row.block ? { ...pick(row.block, ['kind', 'attemptId', 'at']), reason: short(row.block.reason) } : null, + historyRefs: Object.fromEntries(['requirementsHistory', 'waitHistory', 'results', 'admissionDecisions'].map(section => [section, reference(task.id, section, section === 'admissionDecisions' ? row.admission.decisions?.length ?? 0 : row[section]?.length ?? 0)])), + currentRef: reference(task.id, 'requirements'), preview: true } +} +export function taskDetailView(task) { + const result = pick(task, ['id', 'projectId', 'requesterId', 'projectTaskKind', 'status', 'priority', 'progress', 'createdAt', 'updatedAt', 'waitingFor', 'approvalId', 'inputApprovalIds']) + for (const key of ['title', 'criteria', 'note']) if (task[key] !== undefined) result[key] = short(task[key]) + if (task.target) result.target = { ...pick(task.target, ['owner', 'repo', 'number']), href: short(task.target.href) } + if (task.inputRequest) result.inputRequest = { ...pick(task.inputRequest, ['id', 'status']), question: short(task.inputRequest.question) } + if (task.result) result.result = { summary: short(task.result.summary), ...pick(task.result, ['at', 'sha', 'attemptId']) } + result.projectionVersion = 'project-task-detail/v1' + result.source = taskProvenance(task) + result.projectTask = taskContractView(task) + result.historyRefs = Object.fromEntries(['note', 'result', 'actors', 'handoffs'].map(section => [section, reference(task.id, section)])) + return result +} +export function taskHistorySource(task, section) { + if (section === 'admissionDecisions') return task.projectTask?.admission.decisions ?? [] + if (['requirements', 'requirementsHistory', 'waitHistory', 'results', 'dependencies'].includes(section)) return task.projectTask?.[section] ?? (section === 'requirements' ? null : []) + if (section === 'handoffs') return task.collaboration?.history ?? [] + if (['note', 'result', 'actors'].includes(section)) return task[section] ?? null + throw Object.assign(new Error('Unknown scoped Task history section'), { code: 'bot/invalid' }) +} +export function sourcePage(value, body, identity) { + const text = JSON.stringify(value), offset = body.offset ?? 0, limit = body.limit ?? 8000 + if (!Number.isSafeInteger(offset) || offset < 0 || offset > text.length || !Number.isSafeInteger(limit) || limit < 1 || limit > 16000) throw Object.assign(new Error('Invalid source page'), { code: 'bot/invalid' }) + const sha256 = identity(text) + if (body.expectedSha256 !== undefined && body.expectedSha256 !== sha256) throw Object.assign(new Error('Source changed; restart from its current reference'), { code: 'bot/conflict' }) + const next = Math.min(text.length, offset + limit) + return { sha256, offset, text: text.slice(offset, next), totalChars: text.length, nextOffset: next < text.length ? next : null } +} + +export function attemptMetadataView(attempt) { + const result = pick(attempt, ['id', 'protocol', 'projectId', 'taskId', 'workflowId', 'workflowVersion', 'runId', 'status', 'initiatedBy', 'baseSha', 'seedSha', 'candidateSha', 'branch', 'createdAt', 'updatedAt', 'acceptedAt', 'acceptedBy', 'acceptedViaChief', 'cancelRequested', 'supersededBy', 'roleAssignments']) + if (attempt.requirement) result.requirement = { ...pick(attempt.requirement, ['repository', 'projectTaskRevision']), ...Object.fromEntries(['title', 'criteria', 'impact', 'scope', 'note'].filter(key => attempt.requirement[key] !== undefined).map(key => [key, short(attempt.requirement[key])])) } + if (attempt.failure) result.failure = { ...pick(attempt.failure, ['at']), message: short(attempt.failure.message) } + if (attempt.feedback) result.feedback = { ...pick(attempt.feedback, ['previousAttemptId', 'kind', 'candidateSha']), reason: short(attempt.feedback.reason) } + if (attempt.pr) result.pr = pick(attempt.pr, ['number', 'sha', 'url', 'repository']) + if (attempt.publicationBinding) result.publicationBinding = pick(attempt.publicationBinding, ['number', 'repository', 'branch', 'previousSha', 'upstreamSha', 'upstreamRef']) + if (attempt.integrationInputs) result.integrationInputs = attempt.integrationInputs.slice(0, 20).map(row => pick(row, ['version', 'repository', 'ref', 'sha'])) + if (attempt.workspaces) result.workspaces = Object.fromEntries(Object.entries(attempt.workspaces).slice(0, 32).map(([role, row]) => [role, pick(row, ['id', 'status', 'role', 'resourceId'])])) + if (attempt.agentDefinitions) result.agentDefinitions = { revision: attempt.agentDefinitions.revision, roles: (attempt.agentDefinitions.roles ?? []).slice(0, 32).map(row => ({ ...pick(row, ['id', 'role', 'policyRole', 'name', 'model']), skills: (row.skills ?? []).slice(0, 20).map(value => short(value, 300)), execution: pick(row.execution, ['harness', 'model', 'resourceIds']) })) } + function evidence(row) { return { ...pick(row, ['sha', 'sha256', 'passed', 'verdict', 'harness', 'sdkVersion', 'resourceId', 'sessionId', 'model', 'startedAt', 'completedAt']), ...(row.summary ? { summary: short(row.summary) } : {}) } } + result.evidence = {} + for (const [key, row] of Object.entries(attempt.evidence ?? {}).slice(0, 32)) { + if (key === 'execution') result.evidence.execution = Object.fromEntries(Object.entries(row).slice(0, 32).map(([role, value]) => [role, evidence(value)])) + else result.evidence[key] = evidence(row) + } + result.sourceRefs = Object.fromEntries(['evidence', 'feedback', 'requirement', 'failure', 'agentDefinitions', 'integrationAdmissions'].map(section => [section, { query: 'attempt.context', attemptId: attempt.id, section }])) + result.projectionVersion = 'project-attempt-summary/v1' + result.preview = true + return result +} + +// A scoped physical execution fact, not an inference from a missing Bot or PID. +export function executionMetadataView(state, row) { + if (!row) return null + const native = Object.hasOwn(state.projectNativeWorkers ?? {}, row.id) + const attempt = state.projectAttempts?.[row.attemptId] + const session = [attempt?.roleSessions?.[row.role], ...(attempt?.roleSessionHistory ?? [])].find(value => value?.id === row.sessionKey) + const lease = row.leaseId && state.projectResourceLeases?.[row.leaseId] + const exit = native ? nativeExitConfirmed(state, row) : confirmedWorkerExit(state, row) + return { ...pick(row, ['id', 'role', 'resourceId', 'generation', 'sessionKey', 'logicalSessionId', 'createdAt', 'executionStartedAt', 'releasedAt']), + harness: native ? 'flybot' : row.harness ?? null, status: row.status, + policyRole: session?.policyRole ?? row.policyRole ?? row.role ?? null, + definitionRevision: session?.definitionRevision ?? null, + activationEffectId: native ? row.activationEffect?.id ?? null : null, + exitConfirmed: Boolean(exit && !state.bots?.[row.id] && (!row.leaseId || lease?.status === 'released')), + exitProof: row.releaseProof ? pick(row.releaseProof, ['kind', 'generation', 'at', 'registeredWorkEmpty']) : null, + pending: row.pending ? pick(row.pending, ['id', 'kind']) : null, + ...(row.pendingWork ? { pendingWork: row.pendingWork.slice(0, 32).map(value => ({ kind: typeof value.kind === 'string' ? value.kind.slice(0, 80) : 'unknown', count: Number.isSafeInteger(value.count) ? value.count : 0 })) } : {}) } +} diff --git a/core/src/host/store.js b/core/src/host/store.js index 5308f613..459f32aa 100644 --- a/core/src/host/store.js +++ b/core/src/host/store.js @@ -3,6 +3,7 @@ // or group chat it belongs to. import { install as installProjects } from './project-service.js' import { createProjectStateStorage } from './project-state-storage.js' +import { createProjectServiceEvents } from './project-service-events.js' import { constants, mkdirSync, statSync } from 'node:fs' import { copyFile, mkdir, readFile, rename, unlink, writeFile } from 'node:fs/promises' import { join } from 'node:path' @@ -38,6 +39,7 @@ export function install(rt) { // Project services own domain records; store owns their durable snapshot. // projectRuns is a legacy projection; projectExecutions holds direct Run ownership. const state = { version: STATE_VERSION, mainBotIds: [], members: {}, bots: {}, rooms: {}, agents: {}, capabilities: { items: {} }, exchanges: [], exchangeTurns: {}, questions: {}, prefs: {}, invitations: {}, authSessions: {}, tasks: {}, retiredSessions: {}, projects: {}, projectRuns: {}, projectMemory: {}, revision: 0 } + const committedEvents = createProjectServiceEvents(config.projectEventRetention) let loaded = null let writing = Promise.resolve() // Every save this host has accepted since it started — a monotonic watermark, never decremented @@ -397,6 +399,7 @@ export function install(rt) { if (rt.projectEffects?.recover()) migrated = true if (syncDesignatedChief()) migrated = true indexSessions() + committedEvents.load(state) return { migrated: migrated || stale.length > 0, stale } } @@ -558,15 +561,20 @@ export function install(rt) { } } ensureLock() - if (storage.sqlite) storage.write(snapshot) + const prepared = committedEvents.prepare(JSON.parse(snapshot)) + const persistedSnapshot = JSON.stringify(prepared.snapshot, null, 2) + if (storage.sqlite) storage.write(persistedSnapshot) else { - await writeFile(temp, snapshot, 'utf8') + await writeFile(temp, persistedSnapshot, 'utf8') ensureLock() await rename(temp, statePath) } rt.projectEffects?.saved(effectSnapshot) + committedEvents.saved(prepared) + state.projectServiceEventStreams = structuredClone(prepared.snapshot.projectServiceEventStreams) + state.projectServiceEvents = structuredClone(prepared.snapshot.projectServiceEvents) seen = stamp() - await journal?.record(snapshot) + await journal?.record(persistedSnapshot) rt.capabilitiesStateSaved?.() } catch (error) { await unlink(temp).catch(() => { /* the temp file may already be gone */ }) @@ -693,6 +701,7 @@ export function install(rt) { Object.assign(rt, { state, load, save, refresh, readOnly, mutate, savesQueued, savesSettled, + committedProjectEvents: { query: (id, options) => committedEvents.query(id, options), snapshot: (id, options) => committedEvents.snapshot(id, options) }, stateStorage: { status: () => storage.status(), changes: options => storage.changes(options) }, botOf, roomOf, isMain, chatOf, selfOf, isBotSession, chatOwners, groupOwners, pastOwners, diff --git a/core/src/host/tasks.js b/core/src/host/tasks.js index 6c4af1bf..3c3fe7f1 100644 --- a/core/src/host/tasks.js +++ b/core/src/host/tasks.js @@ -1458,6 +1458,8 @@ export function install(rt) { rt.projects?.stampTask(task, body) if (actor?.personalAgentCredentialId) task.personalAgentSource = rt.personalAgentAuth.source(actor, task.projectId) rt.projects?.validateTask(task, body) + const commandSource = task.projectId && rt.projectDevelopment?.commandSource(task.projectId) + if (commandSource) task.commandSource = commandSource if (task.projectId) { const kind = body.projectTaskKind ?? 'code' if (!['code', 'decision', 'release'].includes(kind)) throw invalid('Unknown Project Task kind') diff --git a/core/src/host/workbench-gateway.js b/core/src/host/workbench-gateway.js index 588c58bc..7c0e794b 100644 --- a/core/src/host/workbench-gateway.js +++ b/core/src/host/workbench-gateway.js @@ -1,6 +1,7 @@ // Restricted shared entry. The DSH owner credential stays server-side; members never // receive raw Harness APIs or its owner cookie. Project authorization remains in FlyBot. import { createServer } from 'node:http' +import { createProjectSse } from './project-service-sse.js' import { gzipSync } from 'node:zlib' import { readFile, stat } from 'node:fs/promises' import { resolve } from 'node:path' @@ -25,13 +26,25 @@ export const WORKBENCH_ENDPOINTS = new Set([ const LIMIT = 1024 * 1024 const HTML = 'FlyBot · Project workbench
' -export function createWorkbenchGateway({ upstreamOrigin, upstreamCookie, publicOrigin, readAsset, request = fetch } = {}) { +export function createWorkbenchGateway({ upstreamOrigin, upstreamCookie, publicOrigin, readAsset, request = fetch, eventPollMs = 1000, eventMaxStreams = 64 } = {}) { const upstream = new URL(upstreamOrigin) if (!['http:', 'https:'].includes(upstream.protocol) || !['localhost', '127.0.0.1', '[::1]'].includes(upstream.hostname) || upstream.pathname !== '/' || upstream.search || upstream.hash || upstream.username || upstream.password) throw new Error('Workbench upstream must be a fixed loopback origin') if (typeof upstreamCookie !== 'string' || !upstreamCookie || upstreamCookie.length > 8192 || /[\r\n\u0000]/.test(upstreamCookie)) throw new Error('A valid private upstream cookie is required') const published = publicOrigin ? new URL(publicOrigin) : null if (published && (published.origin !== publicOrigin || published.username || published.password || !['http:', 'https:'].includes(published.protocol))) throw new Error('Invalid fixed workbench public origin') const assets = readAsset ?? (name => readFile(resolve(import.meta.dirname, '../..', name))) + const events = createProjectSse({ pollMs: eventPollMs, maxStreams: eventMaxStreams, + async pull({ projectId, name, payload, token, signal }) { + const response = await request(`${upstream.origin}/api/bot`, { method: 'POST', redirect: 'error', signal: AbortSignal.any([signal, AbortSignal.timeout(10000)]), + headers: { 'content-type': 'application/json', cookie: upstreamCookie, origin: upstream.origin, 'x-flybot-member-session': token }, + body: JSON.stringify({ endpoint: 'project-development', payload: { version: 'project-development/v1', kind: 'query', name, projectId, payload } }) }) + if (!response.ok) throw new Error('Project event runtime is unavailable') + const text = await response.text() + if (text.length > LIMIT) throw new Error('Project event page exceeded its limit') + const result = JSON.parse(text) + if (!result.ok) throw Object.assign(new Error('Project event access/query refused'), { code: result.error?.code }) + return result.value.result + } }) const server = createServer(async (req, res) => { res.setHeader('cache-control', 'no-store') res.setHeader('x-content-type-options', 'nosniff') @@ -53,6 +66,7 @@ export function createWorkbenchGateway({ upstreamOrigin, upstreamCookie, publicO const origin = req.headers.origin if (origin && !allowed.includes(origin) || req.headers['sec-fetch-site'] === 'cross-site') return respond(403, { error: 'Cross-origin request refused' }) const url = new URL(req.url ?? '/', allowed[0]) + if (req.method === 'GET' && url.pathname === '/api/project-events') return events.handle(req, res, url, respond) if (url.search) return respond(404, { error: 'Not found' }) if (req.method === 'GET') { if (url.pathname === '/') return respond(200, HTML, 'text/html; charset=utf-8') @@ -91,6 +105,8 @@ export function createWorkbenchGateway({ upstreamOrigin, upstreamCookie, publicO return respond(200, result) // Never forward upstream Set-Cookie/Location/owner credentials. } catch { if (!res.headersSent) respond(502, { error: 'Workbench runtime request failed' }) } }) + const close = server.close + server.close = function (...args) { events.close(); return close.apply(this, args) } server.on('upgrade', (_request, socket) => { socket.end('HTTP/1.1 403 Forbidden\r\nConnection: close\r\n\r\n') }) return server } @@ -99,7 +115,7 @@ if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1] const value = name => { const index = process.argv.indexOf(name); return index < 0 ? undefined : process.argv[index + 1] } const file = value('--upstream-cookie-file') if (!file || (await stat(file)).mode & 0o077) throw new Error('Use a private mode-0600 upstream cookie file') - const server = createWorkbenchGateway({ upstreamOrigin: value('--upstream'), upstreamCookie: (await readFile(file, 'utf8')).trim(), publicOrigin: value('--public-origin') }) + const server = createWorkbenchGateway({ upstreamOrigin: value('--upstream'), upstreamCookie: (await readFile(file, 'utf8')).trim(), publicOrigin: value('--public-origin'), eventPollMs: Number(value('--event-poll-ms') ?? 1000), eventMaxStreams: Number(value('--event-max-streams') ?? 64) }) const port = Number(value('--port') ?? 3081) server.listen(port, '127.0.0.1', () => process.stdout.write(`Project workbench: http://127.0.0.1:${server.address().port}/\n`)) for (const signal of ['SIGTERM', 'SIGINT']) process.once(signal, () => server.close()) diff --git a/core/test/project-agents.test.js b/core/test/project-agents.test.js index 464ae7fb..319a4c48 100644 --- a/core/test/project-agents.test.js +++ b/core/test/project-agents.test.js @@ -879,3 +879,33 @@ test('role definitions keep exact instruction limits and reject capability/forei await assert.rejects(rt.projectEndpoint('project-agent-definition', { projectId: 'p1', role: 'chief', policyRole: 'developer' }, alice), /capability class/) assert.equal(rt.projectAgents.deliveryDefinitions('p1').revision, revision) }) + +test('versioned role queries and Owner approval reuse original governance and receipt replay', async t => { + const { rt } = await setup(t), initial = await get(rt, alice) + await message(rt, bob, 'public', 'Suggest an analysis role through the service') + const chief = rt.botOf(initial.public.botId) + const proposed = await rt.projectAgents.proposeRole(chief, { requestId: 'v1-analysis-role', action: 'add', role: 'analyst', reason: 'Independent technical analysis supports Project decisions', fields: { name: 'Analyst', policyRole: 'reviewer', instructions: 'Follow the repository Review conventions', skills: ['commit-review'] } }) + const query = { version: 'project-development/v1', kind: 'query', name: 'role.proposals', projectId: 'p1', payload: {} } + const list = await rt.projectEndpoint('project-development', query, bob) + assert.ok(list.result.proposals.some(row => row.id === proposed.proposal.id)) + const approval = { version: 'project-development/v1', kind: 'command', name: 'role.decide', projectId: 'p1', requestId: 'v1-approve-analysis', payload: { proposalId: proposed.proposal.id, expectedSha256: proposed.proposal.sha256, decision: 'approve' } } + await assert.rejects(rt.projectEndpoint('project-development', approval, bob), /access denied|denied/i) + const first = await rt.projectEndpoint('project-development', approval, alice) + assert.equal(first.result.proposal.status, 'approved') + const revision = rt.projectAgents.deliveryDefinitions('p1').revision + const replay = await rt.projectEndpoint('project-development', approval, alice) + assert.equal(replay.replayed, true) + assert.equal(rt.projectAgents.deliveryDefinitions('p1').revision, revision) + const roles = await rt.projectEndpoint('project-development', { ...query, name: 'roles' }, bob) + assert.ok(roles.result.roles.some(row => row.role === 'analyst' && row.policyRole === 'reviewer')) +}) + +test('reading an implicit legacy role catalog does not materialize or mutate Project facts', async t => { + const { rt } = await setup(t) + if (rt.state.projectAgentDefinitions) delete rt.state.projectAgentDefinitions.p1 + const before = JSON.stringify(rt.state.projectAgentDefinitions ?? {}) + const response = await rt.projectEndpoint('project-development', { version: 'project-development/v1', kind: 'query', name: 'roles', projectId: 'p1', payload: {} }, bob) + assert.equal(response.result.revision, 1) + assert.equal(response.result.roles.length, 4) + assert.equal(JSON.stringify(rt.state.projectAgentDefinitions ?? {}), before) +}) diff --git a/core/test/project-command-receipts.test.js b/core/test/project-command-receipts.test.js new file mode 100644 index 00000000..2135e235 --- /dev/null +++ b/core/test/project-command-receipts.test.js @@ -0,0 +1,140 @@ +import assert from 'node:assert/strict' +import { test } from 'node:test' +import { createProjectEffects } from '../src/host/project-effects.js' +import { createCommandReceipts } from '../src/host/project-command-receipts.js' +function fixture(initial) { + const state = structuredClone(initial ?? { members: { alice: {}, bob: {} }, tasks: { t: { id: 't', projectId: 'p', requesterId: 'alice' } } }) + let disk, failSave = false, saves = 0, revoke = false + const rt = { state, readOnly: () => undefined, + requireProjectAccess(actor, projectId, action = 'read') { if (revoke || projectId !== 'p' || !state.members[actor?.memberId] || action !== 'read' && actor.memberId !== 'alice') throw Object.assign(Error('Current access denied'), { code: 'bot/forbidden' }) }, + async save() { ++saves; if (failSave) throw Error('fixture durable failure'); const capture = rt.projectEffects.capture(); disk = structuredClone(state); rt.projectEffects.saved(capture) }, + } + rt.projectEffects = createProjectEffects(rt) + const receipts = createCommandReceipts(rt), actor = { memberId: 'alice' } + const spec = { projectId: 'p', requestId: 'once', name: 'fixture.command', args: { taskId: 't' }, action: 'task-write', taskId: 't', actor } + return { rt, receipts, spec, actor, disk: () => disk, fail: value => { failSave = value }, revoke: () => { revoke = true }, saves: () => saves } +} + +test('concurrent commands commit one result; retries ignore advanced mutable preconditions and reject intent conflicts', async () => { + const f = fixture(); let invoked = 0, preconditions = 0 + const invoke = async () => ({ changed: ++invoked }), check = () => { if (++preconditions > 2) throw Error('stale generation') } + const [first, second] = await Promise.all([f.receipts.run(f.spec, invoke, check), f.receipts.run(f.spec, invoke, check)]) + assert.equal(invoked, 1); assert.deepEqual(first.result, second.result) + const retry = await f.receipts.run(f.spec, invoke, () => { throw Error('advanced generation must not reject a confirmed replay') }) + assert.equal(retry.replayed, true); assert.equal(invoked, 1) + assert.equal(f.rt.projectEffects.intentCommitted(f.rt.state.projectCommandReceipts[first.receipt.id].resultEffect), true) + await assert.rejects(f.receipts.run({ ...f.spec, args: { taskId: 'different' } }, invoke), error => error.code === 'bot/idempotency-conflict') + f.revoke(); await assert.rejects(f.receipts.run(f.spec, invoke), /access denied/) + await f.rt.projectEffects.close() +}) + +test('failed intent cannot dispatch; lost response recovers its durable result after restart without invoking again', async () => { + const f = fixture(); let invoked = 0 + f.fail(true) + await assert.rejects(f.receipts.run(f.spec, async () => ({ changed: ++invoked })), /durable failure/) + assert.equal(invoked, 0); assert.equal(Object.keys(f.rt.state.projectCommandReceipts).length, 0) + f.fail(false) + await f.receipts.run(f.spec, async () => ({ changed: ++invoked })) + const g = fixture(f.disk()); g.rt.projectEffects.recover() + const recovered = await g.receipts.run(g.spec, async () => { throw Error('Must not execute') }) + assert.deepEqual(recovered.result, { changed: 1 }); assert.equal(recovered.replayed, true) + assert.equal(invoked, 1) + await Promise.all([f.rt.projectEffects.close(), g.rt.projectEffects.close()]) +}) + +test('ambiguous command outcomes do not silently repeat and private receipt/effect access stays principal scoped', async () => { + const f = fixture(); let invoked = 0 + await assert.rejects(f.receipts.run(f.spec, async () => { ++invoked; throw Error('External outcome unknown') }), /outcome unknown/) + await assert.rejects(f.receipts.run(f.spec, async () => { ++invoked }), error => error.code === 'bot/outcome-unknown') + assert.equal(invoked, 1) + assert.equal(f.receipts.get('p', 'once', f.actor).status, 'unknown') + assert.throws(() => f.receipts.get('p', 'once', { memberId: 'bob' }), /unavailable/) + assert.equal(f.rt.projectEffects.list('p', { memberId: 'bob' }).length, 0) + const own = f.rt.projectEffects.list('p', f.actor) + assert.ok(own.length >= 2) + await assert.rejects(f.rt.projectEffects.retry({ projectId: 'p', effectId: own[0].id, requestId: 'unauthorized' }, { memberId: 'bob' })) + await f.rt.projectEffects.close() +}) + +test('explicit reconciliation seals a result without a second generic invocation; current Task source is stamped once', async () => { + const f = fixture(); let invoked = 0 + const spec = { ...f.spec, name: 'task.propose' } + await assert.rejects(f.receipts.run(spec, async () => { + ++invoked + const source = f.receipts.source('p') + assert.equal(source.origin.kind, 'human'); assert.ok(source.commandReceiptId) + assert.equal(f.receipts.source('p'), null) + throw Error('response lost after target commit') + })) + const recovered = await f.receipts.run({ ...spec, reconcile: async () => ({ result: { taskId: 'durable-target' } }) }, async () => { ++invoked }) + assert.equal(invoked, 1); assert.equal(recovered.reconstructed, true) + const page = f.receipts.result('p', { commandRequestId: 'once', limit: 1000 }, f.actor) + assert.equal(page.text, JSON.stringify({ taskId: 'durable-target' })) + assert.throws(() => f.receipts.result('p', { commandRequestId: 'once', expectedSha256: 'stale' }, f.actor), /changed/) + await f.rt.projectEffects.close() +}) + +test('an observed result without its final commit is labelled pending and retries finish it without executing twice', async () => { + const f = fixture(); let invoked = 0 + await assert.rejects(f.receipts.run(f.spec, async () => { ++invoked; f.fail(true); return { changed: true } }), /durable failure/) + const pending = f.receipts.get('p', 'once', f.actor) + assert.equal(pending.status, 'completion-pending-commit'); assert.equal(pending.committed, false) + assert.throws(() => f.receipts.result('p', { commandRequestId: 'once' }, f.actor), /not committed/) + f.fail(false) + const recovered = await f.receipts.run(f.spec, async () => { ++invoked }) + assert.equal(invoked, 1); assert.equal(recovered.receipt.committed, true) + assert.deepEqual(recovered.result, { changed: true }) + await f.rt.projectEffects.close() +}) + +test('large recovered results are hash scoped, paged and refuse symlinked payloads', async t => { + const fs = await import('node:fs'), { tmpdir } = await import('node:os'), { join } = await import('node:path') + const f = fixture(); f.rt.home = fs.mkdtempSync(join(tmpdir(), 'flybot-command-result-')) + t.after(async () => { await f.rt.projectEffects.close(); fs.rmSync(f.rt.home, { recursive: true, force: true }) }) + const result = { text: 'result data '.repeat(10000) } + const completed = await f.receipts.run(f.spec, async () => result) + assert.equal((await f.receipts.run(f.spec, async () => { throw Error('No redispatch') })).result.text, result.text) + const row = f.rt.state.projectCommandReceipts[completed.receipt.id] + assert.equal(row.resultText, undefined); assert.equal(row.resultFile, true) + assert.equal(f.receipts.result('p', { commandRequestId: 'once', limit: 99 }, f.actor).text.length, 99) + const file = join(f.rt.home, 'project-command-results', row.resultSha256 + '.json'), outside = join(f.rt.home, 'outside-reference.json') + fs.writeFileSync(outside, JSON.stringify(result)); fs.unlinkSync(file); fs.symlinkSync(outside, file) + assert.throws(() => f.receipts.result('p', { commandRequestId: 'once' }, f.actor), /regular file/) +}) + +test('an omitted candidate precondition differs from explicitly requiring no candidate', async () => { + const f = fixture() + await f.receipts.run(f.spec, async () => ({ changed: true })) + await assert.rejects(f.receipts.run({ ...f.spec, expectedCandidateSha: null }, async () => ({ changed: false })), error => error.code === 'bot/idempotency-conflict') + await f.rt.projectEffects.close() +}) + +test('native and external connections of one independent personal agent share receipts across carrier changes', async () => { + const f = fixture() + f.rt.state.projectAgents = { carrier: { personalAgentId: 'independent-agent' }, replacement: { personalAgentId: 'independent-agent' } } + const native = { memberId: 'alice', agentId: 'carrier', watchGrantId: 'grant-1' } + const external = { memberId: 'alice', agentId: 'independent-agent', personalAgentCredentialId: 'external-credential' } + let invoked = 0 + const first = await f.receipts.run({ ...f.spec, actor: native }, async () => ({ changed: ++invoked })) + assert.equal(first.receipt.origin.kind, 'personal-agent') + assert.equal(first.receipt.origin.agentId, 'independent-agent') + const repeated = await f.receipts.run({ ...f.spec, actor: external }, async () => ({ changed: ++invoked })) + const rotated = await f.receipts.run({ ...f.spec, actor: { ...native, agentId: 'replacement', watchGrantId: 'grant-2' } }, async () => ({ changed: ++invoked })) + assert.equal(invoked, 1); assert.equal(first.receipt.id, repeated.receipt.id); assert.equal(rotated.receipt.id, first.receipt.id) + f.rt.state.members.alice.status = 'disabled' + await assert.rejects(f.receipts.run({ ...f.spec, actor: external }, async () => ({})), /disabled/) + await f.rt.projectEffects.close() +}) + +test('request IDs are operation scoped; ambiguous lookups require a name and missing receipt queries are pure', async () => { + const f = fixture(), original = JSON.stringify(f.rt.state) + assert.throws(() => f.receipts.get('p', 'missing', f.actor), /unavailable/) + assert.equal(JSON.stringify(f.rt.state), original) + const first = await f.receipts.run(f.spec, async () => ({ first: true })) + const second = await f.receipts.run({ ...f.spec, name: 'fixture.other' }, async () => ({ second: true })) + assert.notEqual(first.receipt.id, second.receipt.id) + assert.throws(() => f.receipts.get('p', 'once', f.actor), error => error.code === 'bot/ambiguous-command') + assert.equal(f.receipts.get('p', 'once', f.actor, 'fixture.other').id, second.receipt.id) + assert.equal(f.receipts.result('p', { commandRequestId: 'once', commandName: f.spec.name }, f.actor).text, '{"first":true}') + await f.rt.projectEffects.close() +}) diff --git a/core/test/project-delivery.test.js b/core/test/project-delivery.test.js index 7f162680..c8fc9656 100644 --- a/core/test/project-delivery.test.js +++ b/core/test/project-delivery.test.js @@ -1185,20 +1185,22 @@ test('an approved extra merge seals when the published upstream is already an an assert.equal(f.attempt.pr,undefined) }) -const integrationCall = (f, kind, name, payload, actor = alice, requestId = 'approve-backend') => f.rt.projectEndpoint('project-development', { version: 'project-development/v1', kind, name, projectId: 'p1', ...(kind === 'command' ? { requestId } : {}), payload: { attemptId: f.attempt.id, ...payload } }, actor) +const integrationCall = (f, kind, name, payload, actor = alice, requestId = `integration-${f.integrationSequence = (f.integrationSequence ?? 0) + 1}`) => f.rt.projectEndpoint('project-development', { version: 'project-development/v1', kind, name, projectId: 'p1', ...(kind === 'command' ? { requestId } : {}), payload: { attemptId: f.attempt.id, ...payload } }, actor) const indexBytes = f => createHash('sha256').update(readFileSync(git(f.path,['rev-parse','--git-path','index']))).digest('hex') test('manager explicitly pins integration input through service snapshot without changing index or approving delivery', async t => { const f=await additionalIntegrationFixture(t),before=indexBytes(f) const snapshot=(await integrationCall(f,'query','attempt.integration-state',{})).result const payload={inputs:[{ref:f.ref,sha:f.source}],generation:snapshot.generation,expectedWorkspace:snapshot.snapshot} - const approved=await integrationCall(f,'command','attempt.integration-bind',payload) + const approved=await integrationCall(f,'command','attempt.integration-bind',payload,alice,'approve-backend') assert.equal(approved.result.already,false) assert.equal(approved.result.admission.generation,snapshot.generation+1) assert.equal(indexBytes(f),before) assert.equal(git(f.path,['rev-parse','MERGE_HEAD']),f.source) const reordered={...payload,expectedWorkspace:Object.fromEntries(Object.entries(payload.expectedWorkspace).reverse())} - assert.equal((await integrationCall(f,'command','attempt.integration-bind',reordered)).result.already,true) + const replay=await integrationCall(f,'command','attempt.integration-bind',reordered,alice,'approve-backend') + assert.equal(replay.replayed,true) + assert.equal(replay.result.admission.id,approved.result.admission.id) assert.equal(f.attempt.integrationAdmissions.length,1) assert.equal(f.attempt.candidateSha,undefined) const node=await f.rt.projectDelivery.preparePoolRole(f.task,{id:'resumed-integration',attemptId:f.attempt.id,generation:99},f.developer) @@ -1335,9 +1337,9 @@ test('approved object availability alone cannot replace a real merge or reuse an const f=await additionalIntegrationFixture(t,{merge:false}) const state=(await integrationCall(f,'query','attempt.integration-state',{})).result const payload={inputs:[{ref:f.ref,sha:f.source}],generation:state.generation,expectedWorkspace:state.snapshot} - await integrationCall(f,'command','attempt.integration-bind',payload) + await integrationCall(f,'command','attempt.integration-bind',payload,alice,'reuse-source') const before=indexBytes(f) - await assert.rejects(integrationCall(f,'command','attempt.integration-bind',{...payload,generation:state.generation+1}),/requestId was reused/) + await assert.rejects(integrationCall(f,'command','attempt.integration-bind',{...payload,generation:state.generation+1},alice,'reuse-source'),/requestId was reused/) await assert.rejects(f.rt.projectDelivery.completePoolRole({task:f.task,claim:{attemptId:f.attempt.id},role:'developer',outputs:{}}),/merge every pinned integration/) assert.equal(indexBytes(f),before) assert.equal(git(f.path,['rev-parse','HEAD']),f.seed) @@ -1531,6 +1533,8 @@ test('a parent cannot seal skipped or subsequently invalidated child integration test('approved custom role capabilities preserve logical identities while sealing testing and reviewing real code', async t => { const f = await setup(t, undefined, { pool: true }) f.rt.projectPools.connect({ validate: () => ({ ok: true }) }) + // Explicit setup is a write; definition metadata reads do not materialize catalogs. + await f.rt.projectEndpoint('project-agent-definition', { projectId: 'p1', role: 'developer', name: 'Developer' }, alice) const original = f.rt.projectAgents.deliveryDefinitions('p1') const names = { developer: 'implementer', tester: 'validator', reviewer: 'auditor' } // Governance approval is tested through real host APIs separately. This fixture diff --git a/core/test/project-development-service.test.js b/core/test/project-development-service.test.js index 0a41059e..28429308 100644 --- a/core/test/project-development-service.test.js +++ b/core/test/project-development-service.test.js @@ -132,7 +132,7 @@ test('trusted role and schedule ports retain original claims, authority and even test('proposal/worker targets stay in envelope scope and permission request identity is preserved', async () => { const { service, state, rt, actor } = fixture() state.projectAgentProposals = { foreign: { projectId: 'other', memberId: 'alice' } } - state.projectHarnessWorkers = { foreign: { projectId: 'other' }, own: { projectId: 'p' } } + state.projectHarnessWorkers = { foreign: { projectId: 'other' }, own: { projectId: 'p', taskId: 't' } } let calls = 0 rt.projectAgents.endpoint = async () => { calls++; return {} } rt.projectHarness = { input: async body => { calls++; assert.equal(body.requestId, 'permission-request'); return { sent: true } } } @@ -213,3 +213,96 @@ test('publication recovery shows its actual wait and stale outbox candidates can task.chiefQuestion = { reason: 'Human scope decision' } assert.equal(taskView(state, task).phase, 'awaiting-owner') }) + +test('Task summaries bound historical context and never expose execution requester sessions', async () => { + const { service, state, actor } = fixture(), task = state.tasks.t + task.title = 'Current requirement' + task.projectTask = { version: 'project-task/v1', admission: { status: 'accepted' }, requirements: { revision: 7, title: task.title, criteria: 'Regression passes', impact: 'Improve the Project', scope: 'Current scope' }, dependencies: [], + requirementsHistory: Array.from({ length: 120 }, (_, n) => ({ revision: n, scope: 'historical scope '.repeat(2000) })), waitHistory: [], results: [], executionRequester: { memberId: 'alice', sessionId: 'private-requester-session' } } + const before = JSON.stringify(state) + const view = service.view(task, actor) + assert.ok(Buffer.byteLength(JSON.stringify(view)) < 16000) + assert.equal(JSON.stringify(view).includes('private-requester-session'), false) + assert.equal(view.contract.requirements.revision, 7) + assert.equal(view.contract.historyRefs.requirementsHistory.count, 120) + const query = { version: DEVELOPMENT_PROTOCOL, kind: 'query', name: 'task.history', projectId: 'p', payload: { taskId: 't', section: 'requirementsHistory', limit: 16000 } } + const first = (await service.endpoint('project-development', query, actor)).result + assert.equal(first.text.length, 16000) + assert.equal(first.text, JSON.stringify(task.projectTask.requirementsHistory).slice(0, 16000)) + await assert.rejects(service.endpoint('project-development', { ...query, payload: { ...query.payload, expectedSha256: 'stale' } }, actor), /changed/) + await assert.rejects(service.endpoint('project-development', { ...query, payload: { ...query.payload, section: 'executionRequester' } }, actor)) + await assert.rejects(service.endpoint('project-development', { ...query, projectId: 'other' }, actor)) + assert.equal(JSON.stringify(state), before) +}) + +test('Attempt summaries bound repeated sessions, role definitions and evidence while exact scoped sources remain readable', async () => { + const { service, state, actor } = fixture(), huge = 'long role report '.repeat(10000) + state.projectAttempts.large = { id: 'large', projectId: 'p', taskId: 't', createdAt: 1, status: 'failed', failure: { message: huge }, feedback: { reason: huge }, requirement: { title: huge, criteria: huge }, + roleSessionHistory: Array.from({ length: 200 }, (_, n) => ({ id: 'session-' + n, role: 'developer', context: { text: huge, sha256: 'hash' }, policy: { text: huge, sha256: 'policy' } })), + agentDefinitions: { roles: [{ id: 'p:developer', role: 'developer', persona: huge, memory: huge, execution: { harness: 'codex' } }] }, evidence: { review: { sha: 'commit', verdict: 'approve', summary: huge } } } + const before = JSON.stringify(state) + const detail = await service.endpoint('project-task-detail', { taskId: 't' }, actor) + assert.ok(Buffer.byteLength(JSON.stringify(detail)) < 50000) + const summary = detail.attempts[0] + assert.equal(summary.roleSessionHistory.length, 3) + assert.equal(summary.roleSessionHistoryPage.total, 200) + assert.equal(summary.agentDefinitions.roles[0].persona, undefined) + const query = { version: DEVELOPMENT_PROTOCOL, kind: 'query', name: 'attempt.context', projectId: 'p', payload: { attemptId: 'large', section: 'evidence', limit: 16000 } } + assert.equal((await service.endpoint('project-development', query, actor)).result.text, JSON.stringify(state.projectAttempts.large.evidence).slice(0, 16000)) + assert.equal(JSON.stringify(state), before) +}) + +test('durable v1 command replays its confirmed response after generation advances and keeps source keys distinct', async () => { + const { createProjectEffects } = await import('../src/host/project-effects.js') + const { rt, state, actor } = fixture() + state.tasks.t.collaboration = { generation: 4, history: [] } + rt.projectEffects = createProjectEffects(rt) + rt.save = async () => rt.projectEffects.saved(rt.projectEffects.capture()) + let starts = 0 + rt.projectScheduler.request = async () => { ++starts; state.tasks.t.collaboration.generation = 5; return { queue: { id: 'one-queue', projectId: 'p' } } } + const service = createDevelopmentService(rt) + const request = { version: DEVELOPMENT_PROTOCOL, kind: 'command', name: 'task.run', projectId: 'p', requestId: 'stable-command', expectedGeneration: 4, payload: { taskId: 't' } } + const first = await service.endpoint('project-development', request, actor) + const repeated = await service.endpoint('project-development', request, actor) + assert.equal(starts, 1) + assert.equal(first.receipt.committed, true); assert.equal(repeated.replayed, true) + assert.deepEqual(first.result, repeated.result) + await assert.rejects(service.endpoint('project-development', { ...request, expectedGeneration: 5 }, actor), error => error.code === 'bot/idempotency-conflict') + const query = { version: DEVELOPMENT_PROTOCOL, kind: 'query', name: 'command.receipt', projectId: 'p', payload: { commandRequestId: 'stable-command' } } + assert.equal((await service.endpoint('project-development', query, actor)).result.id, first.receipt.id) + await assert.rejects(service.endpoint('project-development', query, { memberId: 'bob' }), /unavailable/) + await rt.projectEffects.close() +}) + +test('native execution views distinguish committed activation, current roles and confirmed release without host internals', () => { + const { state } = fixture(), task = state.tasks.t + task.deliveryAttemptId = 'a'; task.collaboration = { status: 'claimed', botId: 'p:implementer', generation: 2, history: [], claim: { id: 'claim', workerId: 'w', role: 'implementer', policyRole: 'developer', generation: 2, status: 'active' } } + state.projectAgentDefinitions = { p: { revision: 2, roles: [] } } + state.projectAttempts.a = { id: 'a', projectId: 'p', taskId: 't', status: 'running', roleSessions: { implementer: { id: 'logical', definitionRevision: 1 } } } + state.bots.w = { id: 'w', projectId: 'p', attemptId: 'a' } + const worker = { id: 'w', projectId: 'p', taskId: 't', attemptId: 'a', role: 'implementer', sessionKey: 'logical', generation: 2, version: 'project-native-worker/v2', status: 'bound', sessionId: 'physical', bindingKey: 'binding', activationEffect: { id: 'activate' }, processScope: { unit: 'HIDDEN_HOST_DETAIL' }, failure: 'HIDDEN_HOST_DETAIL' } + state.projectNativeWorkers = { w: worker } + const view = options => taskView(state, task, { activity: { w: 'thinking' }, ...options }) + assert.equal(view({ isIntentCommitted: () => false }).phase, 'starting') + assert.equal(view({ isIntentCommitted: () => true }).phase, 'running') + assert.equal(view().currentRole, 'implementer') + assert.equal(view().roleDefinitionRevision, 2) + assert.equal(view().execution.definitionRevision, 1) + assert.equal(JSON.stringify(view()).includes('HIDDEN_HOST_DETAIL'), false) + worker.status = 'stop-unconfirmed' + assert.equal(view().waitingFor, 'worker-exit') + assert.equal(view().execution.exitConfirmed, false) + worker.status = 'quiescent'; worker.releaseProof = { kind: 'native-agent-quiescent', registeredWorkEmpty: true, workerId: 'w', sessionId: 'physical', bindingKey: 'binding', generation: 2 } + delete state.bots.w + assert.equal(view().execution.exitConfirmed, true) +}) + +test('attempt lists reference large frozen definitions instead of copying their content into each session', async () => { + const { service, rt, state, actor } = fixture(), text = 'frozen persona source '.repeat(20000) + state.projectAttempts.a = { id: 'a', projectId: 'p', taskId: 't', roleSessions: { developer: { id: 'session', role: 'developer', definitionRef: 'original-definition', definitionRevision: 1, definition: { instructions: text, resolvedSkills: [{ content: text }] } } } } + rt.projectDelivery = { endpoint: async () => ({ task: state.tasks.t, attempts: [state.projectAttempts.a] }) } + const result = await service.endpoint('project-task-detail', { taskId: 't' }, actor) + assert.ok(JSON.stringify(result).length < 16000) + assert.equal(result.attempts[0].roleSessions.developer.definitionRevision, 1) + assert.equal(state.projectAttempts.a.roleSessions.developer.definition.instructions, text) +}) diff --git a/core/test/project-effects-runtime.test.js b/core/test/project-effects-runtime.test.js index 8afcc15b..661d72a6 100644 --- a/core/test/project-effects-runtime.test.js +++ b/core/test/project-effects-runtime.test.js @@ -60,7 +60,7 @@ test('v1 Task input commits the inbox and notify intent before returning and cre const disk = JSON.parse(readFileSync(join(f.home, 'state.json'), 'utf8')) assert.equal(disk.projectTaskInputs[first.result.input.id].text, command.payload.text) assert.ok(Object.values(disk.projectEffects).some(row => row.kind === 'input.notify' && row.payload.inputId === first.result.input.id)) - assert.equal((await f.rt.projectEndpoint('project-development', command, { memberId: 'owner-1' })).result.already, true) + assert.equal((await f.rt.projectEndpoint('project-development', command, { memberId: 'owner-1' })).replayed, true) const list = await f.rt.projectEndpoint('project-development', { version: 'project-development/v1', kind: 'query', name: 'inputs', projectId: 'p', payload: { taskId: 'task' } }, { memberId: 'owner-1' }) assert.equal(list.result.inputs.length, 1) assert.equal(Object.keys(f.rt.state.tasks).length, 1) @@ -89,7 +89,9 @@ test('v1 GitHub subscription polling commits source cursor and original Task inp assert.equal(disk.projectThreads[sub.threadId].cursor.pages.comments,1) assert.ok(Object.values(disk.projectEffects).some(row=>row.kind==='input.notify'&&row.payload.inputId===input.id)) assert.equal(Object.keys(disk.tasks).length,1) - assert.equal((await command('source.poll',{threadId:sub.threadId})).result.inputs,0) + assert.equal((await command('source.poll',{threadId:sub.threadId})).replayed,true) + const fresh=await f.rt.projectEndpoint('project-development',{version:'project-development/v1',kind:'command',projectId:'p',name:'source.poll',requestId:'fresh-poll',payload:{threadId:sub.threadId}},actor) + assert.equal(fresh.result.inputs,0) }) test('scheduled Source polling survives closed browser sessions using scoped current-principal broker proof',async t=>{ @@ -119,6 +121,9 @@ test('real membership removal cancels its pending input but preserves the Projec assert.ok(f.rt.state.tasks.task) await assert.rejects(f.rt.projectEndpoint('project-development',command,{memberId:'bob'}),/access denied/) f.rt.state.projects.p.roles.members.push('bob') - assert.equal((await f.rt.projectEndpoint('project-development',command,{memberId:'bob'})).result.already,true) + assert.equal((await f.rt.projectEndpoint('project-development',command,{memberId:'bob'})).replayed,true) assert.equal(f.rt.state.projectTaskInputs[published.result.input.id].status,'cancelled') + const domain=await f.rt.projectInputs.publish({...command.payload,projectId:'p',requestId:command.requestId},{memberId:'bob'}) + assert.equal(domain.already,true) + assert.equal(domain.input.status,'cancelled') }) diff --git a/core/test/project-pools.test.js b/core/test/project-pools.test.js index 81abd880..dfbe700e 100644 --- a/core/test/project-pools.test.js +++ b/core/test/project-pools.test.js @@ -520,6 +520,28 @@ test('a sealed parent resumes through one successor Attempt while retaining its await f.rt.projectEffects.close() }) +test('failed durable acceptance cannot be replayed as a claim; retries commit one active owner', async () => { + const f = await projectTaskFixture() + await f.admit() + const handoffId = f.task.collaboration.currentHandoffId, save = f.rt.save + f.rt.save = async () => { throw new Error('fixture acceptance commit failed') } + await assert.rejects(f.pool.accept('task', 'p:developer', handoffId, 'accept-once'), /commit failed/) + assert.equal(f.task.collaboration.status, 'queued') + assert.equal(f.task.collaboration.claim, undefined) + assert.equal(f.task.collaboration.history.at(-1).acceptance, undefined) + f.rt.save = save + const [first, duplicate] = await Promise.all([ + f.pool.accept('task', 'p:developer', handoffId, 'accept-once'), + f.pool.accept('task', 'p:developer', handoffId, 'accept-once'), + ]) + assert.equal(first.id, duplicate.id) + assert.ok(first.acceptanceEffect) + assert.equal(f.rt.projectEffects.intentCommitted(first.acceptanceEffect), true) + assert.equal(f.task.collaboration.history.at(-1).acceptance.effect.id, first.acceptanceEffect.id) + assert.equal(f.task.collaboration.claim.id, first.id) + await f.rt.projectEffects.close() +}) + test('a retired source can finish its frozen active claim but cannot receive new offers', async () => { const f = setup(); await f.admit(); const claim = await f.claim() diff --git a/core/test/project-service-events.test.js b/core/test/project-service-events.test.js new file mode 100644 index 00000000..ec6de910 --- /dev/null +++ b/core/test/project-service-events.test.js @@ -0,0 +1,101 @@ +import assert from 'node:assert/strict' +import { test } from 'node:test' +import { createProjectServiceEvents } from '../src/host/project-service-events.js' +const seed = () => ({ projects: { p: { id: 'p', status: 'active', goal: 'Agreed goal' }, other: { id: 'other', status: 'active' } }, tasks: { task: { id: 'task', projectId: 'p', title: 'Current work', status: 'open' } } }) + +test('only acknowledged snapshots expose events; scoped cursors reconnect across restart', () => { + const bus = createProjectServiceEvents(), state = seed() + bus.load(state) + const start = bus.snapshot('p') + const next = structuredClone(state); next.tasks.task.status = 'working' + const pending = bus.prepare(next) + assert.equal(bus.query('p', { after: start.cursor }).events.length, 0) + bus.saved(pending) + const page = bus.query('p', { after: start.cursor }) + assert.equal(page.events.length, 1); assert.equal(page.events[0].data.status, 'working') + assert.equal(bus.query('p', { after: page.next }).events.length, 0) + assert.equal(bus.query('other').events.length, 0) + assert.throws(() => bus.query('other', { after: page.next }), /scope/) + const restarted = createProjectServiceEvents(); restarted.load(pending.snapshot) + assert.deepEqual(restarted.query('p', { after: start.cursor }), page) +}) + +test('uncommitted changes and private records cannot leak or perturb public Project event sequence', () => { + const bus = createProjectServiceEvents(), state = seed(); bus.load(state) + const initial = bus.snapshot('p') + state.projectAgents = { private: { notes: ['private-secret'], ownerUserId: 'owner', projectId: 'p' } } + state.projectConsultations = { private: { text: 'private-secret', projectId: 'p' } } + state.authSessions = { sensitive: { token: 'secret-token' } } + const privateSave = bus.prepare(structuredClone(state)); bus.saved(privateSave) + assert.equal(bus.snapshot('p').cursor, initial.cursor) + assert.equal(JSON.stringify(bus.query('p')).includes('private-secret'), false) + const candidate = structuredClone(state); candidate.tasks.task.status = 'accepted' + bus.prepare(candidate) // failed storage write; never acknowledged + assert.equal(bus.snapshot('p').tasks[0].status, 'open') + const recovered = bus.prepare(structuredClone(state)); bus.saved(recovered) + assert.equal(bus.query('p').events.length, 0) +}) + +test('bounded retention expires old cursors and snapshots require a consistent revision for later pages', () => { + let clock = 1000 + const bus = createProjectServiceEvents({ maxPerProject: 2, maxAgeMs: 1000, now: () => clock }), state = seed(); bus.load(state) + const old = bus.snapshot('p').cursor + for (let n = 0; n < 3; n++) { const next = structuredClone(state); next.tasks.task.status = 'step-' + n; bus.saved(bus.prepare(next)) } + assert.equal(bus.query('p').events.length, 2) + assert.throws(() => bus.query('p', { after: old }), error => error.code === 'bot/cursor-expired' && error.nextAction === 'project.snapshot') + assert.throws(() => bus.snapshot('p', { expectedRevision: 0 }), /changed/) + clock += 2000 + bus.saved(bus.prepare(structuredClone(state))) + assert.equal(bus.query('p').events.length, 1, 'only the new changed state survives age retention') + for (const options of [{ limit: 0 }, { limit: 101 }, { after: 'malformed' }]) assert.throws(() => bus.query('p', options)) +}) + +test('clock rollback cannot create a retention hole; malformed persisted streams fail closed', () => { + let clock = 5000 + const bus = createProjectServiceEvents({ maxPerProject: 3, maxAgeMs: 1000, now: () => clock }), state = seed(); bus.load(state) + let prepared + for (const [at, status] of [[5000, 'one'], [1000, 'two'], [5100, 'three']]) { clock = at; state.tasks.task.status = status; prepared = bus.prepare(structuredClone(state)); bus.saved(prepared) } + assert.deepEqual(bus.query('p').events.map(row => row.at), [5000, 5000, 5100]) + const restarted = createProjectServiceEvents(); restarted.load(prepared.snapshot) + const broken = structuredClone(prepared.snapshot), rows = Object.values(broken.projectServiceEvents) + delete broken.projectServiceEvents[Object.keys(broken.projectServiceEvents)[0]] + assert.throws(() => restarted.load(broken), /gap|duplicate/) + const duplicate = structuredClone(prepared.snapshot) + duplicate.projectServiceEvents.duplicate = rows[0] + assert.throws(() => restarted.load(duplicate), /gap|duplicate/) +}) + +test('committed Project facts include current role revision and native activation/exit facts without host data', () => { + const bus = createProjectServiceEvents(), state = seed() + state.projectAgentDefinitions = { p: { revision: 1, roles: [] } } + state.projectAttempts = { a: { id: 'a', projectId: 'p', taskId: 'task' } } + bus.load(state) + assert.equal(bus.snapshot('p').project.roleDefinitionRevision, 1) + const cursor = bus.snapshot('p').cursor, next = structuredClone(state) + next.projectAgentDefinitions.p.revision = 2 + next.projectNativeWorkers = { native: { id: 'native', projectId: 'p', taskId: 'task', attemptId: 'a', version: 'project-native-worker/v2', role: 'developer', status: 'bound', generation: 2, sessionKey: 'session', sessionId: 'physical', activationEffect: { id: 'activate' }, processScope: { unit: 'HIDDEN_HOST_DETAIL' } } } + const pending = bus.prepare(next) + assert.equal(bus.query('p', { after: cursor }).events.length, 0) + bus.saved(pending) + const events = bus.query('p', { after: cursor }).events + assert.equal(events.find(row => row.entityType === 'project').data.roleDefinitionRevision, 2) + assert.equal(events.find(row => row.entityType === 'execution').data.harness, 'flybot') + assert.equal(events.find(row => row.entityType === 'execution').data.activationEffectId, 'activate') + assert.equal(JSON.stringify(events).includes('HIDDEN_HOST_DETAIL'), false) +}) + +test('committed role approval proposals are public scoped facts without personal proposal contents', () => { + const bus = createProjectServiceEvents(), state = seed(); bus.load(state) + const cursor = bus.snapshot('p').cursor, next = structuredClone(state) + next.projectRoleProposals = { proposal: { id: 'proposal', projectId: 'p', action: 'add', role: 'analyst', status: 'pending', baseRevision: 1, definition: { instructions: 'HIDDEN_DEFINITION' } } } + next.projectAgentProposals = { personal: { id: 'personal', projectId: 'p', text: 'HIDDEN_PERSONAL_PROPOSAL' } } + const pending = bus.prepare(next); bus.saved(pending) + const events = bus.query('p', { after: cursor }).events + assert.equal(events.length, 1) + assert.equal(events[0].entityType, 'role-proposal') + assert.equal(events[0].data.role, 'analyst') + assert.equal(JSON.stringify(events).includes('HIDDEN_'), false) + const snapshot = bus.snapshot('p', { entityType: 'role-proposal' }) + assert.deepEqual(snapshot.entities.map(row => row.id), ['proposal']) + assert.throws(() => bus.snapshot('p', { entityType: 'private-agent' }), /Unknown public/) +}) diff --git a/core/test/project-service-sse.test.js b/core/test/project-service-sse.test.js new file mode 100644 index 00000000..55cf2d8c --- /dev/null +++ b/core/test/project-service-sse.test.js @@ -0,0 +1,93 @@ +// Actual HTTP/SSE transport with explicit in-memory event/authorization fixtures. +import assert from 'node:assert/strict' +import { createServer } from 'node:http' +import { once } from 'node:events' +import { test } from 'node:test' +import { createWorkbenchGateway } from '../src/host/workbench-gateway.js' +import { createProjectServiceEvents } from '../src/host/project-service-events.js' +async function fixture(t, maxStreams = 4) { + const state = { projects: { p: { id: 'p', status: 'active' } }, tasks: { task: { id: 'task', projectId: 'p', status: 'open' } } }, seen = [] + let bus = createProjectServiceEvents({ maxPerProject: 2 }), allowed = true + bus.load(state) + const upstream = createServer(async (req, res) => { + let text = ''; for await (const chunk of req) text += chunk + const body = JSON.parse(text), input = body.payload + seen.push({ token: req.headers['x-flybot-member-session'], cookie: req.headers.cookie, name: input.name }) + let result + try { + if (!allowed || req.headers['x-flybot-member-session'] !== 'fixture-member') throw Object.assign(new Error('Fixture revoked'), { code: 'bot/forbidden' }) + if (body.endpoint !== 'project-development' || input.kind !== 'query' || input.projectId !== 'p') throw Error('Wrong service query') + result = { ok: true, value: { result: input.name === 'events' ? bus.query(input.projectId, input.payload) : bus.snapshot(input.projectId, input.payload) } } + } catch (error) { result = { ok: false, error: { code: error.code ?? 'bot/error', details: error.details } } } + res.writeHead(200, { 'content-type': 'application/json', 'set-cookie': 'private-owner=do-not-forward' }); res.end(JSON.stringify(result)) + }) + upstream.listen(0, '127.0.0.1'); await once(upstream, 'listening') + const gateway = createWorkbenchGateway({ upstreamOrigin: `http://127.0.0.1:${upstream.address().port}`, upstreamCookie: 'owner=private-owner', eventPollMs: 10, eventMaxStreams: maxStreams }) + gateway.listen(0, '127.0.0.1'); await once(gateway, 'listening') + const base = `http://127.0.0.1:${gateway.address().port}` + t.after(async () => { gateway.closeAllConnections(); upstream.closeAllConnections(); await Promise.all([new Promise(done => gateway.close(done)), new Promise(done => upstream.close(done))]) }) + function update(status) { state.tasks.task.status = status; const prepared = bus.prepare(structuredClone(state)); bus.saved(prepared); return prepared.snapshot } + return { base, seen, gateway, state, bus: () => bus, update, revoke: () => { allowed = false }, restart(snapshot) { bus = createProjectServiceEvents({ maxPerProject: 2 }); bus.load(snapshot) }, + open: (headers = {}, signal) => fetch(`${base}/api/project-events?projectId=p`, { headers: { 'x-flybot-member-session': 'fixture-member', ...headers }, signal: signal ? AbortSignal.any([signal, AbortSignal.timeout(5000)]) : AbortSignal.timeout(5000) }) } +} +async function readUntil(reader, match, initial = '') { + let text = initial + while (!match(text)) { const next = await reader.read(); if (next.done) break; text += new TextDecoder().decode(next.value) } + return text +} + +test('SSE uses the original authenticated query service and reauthenticates before subsequent facts', async t => { + const f = await fixture(t), response = await f.open() + assert.equal(response.status, 200); assert.match(response.headers.get('content-type'), /text\/event-stream/) + assert.equal(response.headers.get('set-cookie'), null) + const reader = response.body.getReader(), first = await readUntil(reader, text => text.includes('event: snapshot')) + f.revoke(); f.update('after-revocation') + const all = await readUntil(reader, () => false, first) + assert.ok(all.includes('event: access-revoked')) + assert.equal(all.includes('after-revocation'), false) + assert.equal(all.includes('private-owner'), false) + assert.ok(f.seen.every(row => row.token === 'fixture-member' && row.cookie === 'owner=private-owner')) + assert.deepEqual(f.seen.slice(0, 2).map(row => row.name), ['project.snapshot', 'events']) +}) + +test('SSE Last-Event-ID resumes ordered committed facts after restart without replaying the acknowledged event', async t => { + const f = await fixture(t), start = f.bus().snapshot('p').cursor + f.update('working'); const snapshot = f.update('blocked') + f.restart(snapshot) + const control = new AbortController(), response = await f.open({ 'last-event-id': start }, control.signal), reader = response.body.getReader() + const first = await readUntil(reader, text => (text.match(/event: fact/g) ?? []).length >= 2) + const facts = [...first.matchAll(/data: (.+)\n/g)].map(match => JSON.parse(match[1])).filter(row => row.type) + assert.deepEqual(facts.map(row => row.data.status), ['working', 'blocked']) + control.abort() + const secondControl = new AbortController(), resumed = await f.open({ 'last-event-id': facts[0].cursor }, secondControl.signal) + const second = await readUntil(resumed.body.getReader(), text => text.includes('event: fact')) + assert.equal(second.includes('"status":"working"'), false) + assert.ok(second.includes('"status":"blocked"')) + secondControl.abort() +}) + +test('expired SSE cursors require snapshots; credentials in URLs, missing headers and cross-scope IDs are refused', async t => { + const f = await fixture(t), old = f.bus().snapshot('p').cursor + for (const status of ['one', 'two', 'three']) f.update(status) + const expired = await f.open({ 'last-event-id': old }) + assert.equal(expired.status, 409) + assert.equal((await expired.json()).error.details.nextAction, 'project.snapshot') + const missing = await fetch(`${f.base}/api/project-events?projectId=p`) + assert.equal(missing.status, 401) + assert.equal((await fetch(`${f.base}/api/project-events?projectId=p&token=secret`)).status, 400) + const foreign = JSON.parse(Buffer.from(f.bus().snapshot('p').cursor, 'base64url').toString('utf8')) + foreign.p = 'another-project' + const cross = await f.open({ 'last-event-id': Buffer.from(JSON.stringify(foreign)).toString('base64url') }) + assert.equal(cross.status, 409) + assert.equal((await cross.json()).error.code, 'bot/conflict') +}) + +test('SSE connections are bounded and server shutdown aborts open streams', async t => { + const f = await fixture(t, 1), response = await f.open(), reader = response.body.getReader() + await readUntil(reader, text => text.includes('event: snapshot')) + const over = await f.open() + assert.equal(over.status, 429); assert.equal((await over.json()).error.code, 'bot/busy') + const closed = once(f.gateway, 'close'); f.gateway.close() + await readUntil(reader, () => false) + await closed +}) diff --git a/core/test/project-sources.test.js b/core/test/project-sources.test.js index 92265eab..61df34c6 100644 --- a/core/test/project-sources.test.js +++ b/core/test/project-sources.test.js @@ -19,7 +19,11 @@ function fixture(state) { rt.projectEffects = createProjectEffects(rt); inputs(rt); sources(rt); rt.projectDevelopment = createDevelopmentService(rt) return { rt, project, schedules, disk: () => disk, reads: () => reads, fail(value) { failure = value }, revoke() { revoked = true }, feed(value) { feed = value } } } -const command = (f, name, payload, actor = owner, requestId = 'request') => f.rt.projectDevelopment.endpoint('project-development', { version: 'project-development/v1', kind: 'command', name, projectId: 'p', requestId, payload }, actor) +// Domain/fault tests use the original aliases on the same facade so receipt-intent +// saves cannot consume the injected source-transaction failure. Real v1 source and +// response-loss recovery are covered in project-effects-runtime.test.js. +const sourceEndpoints = { 'source.subscribe': 'project-source-subscribe', 'source.poll': 'project-source-poll', 'source.disable': 'project-source-disable' } +const command = async (f, name, payload, actor = owner, requestId = 'request') => ({ result: await f.rt.projectDevelopment.endpoint(sourceEndpoints[name], { ...payload, projectId: 'p', requestId }, actor) }) const subscribe = async (f, payload = {}) => (await command(f, 'source.subscribe', { taskId: 't', threadKind: 'pull', number: 7, includeHistory: true, ...payload })).result.subscription const poll = (f, sub, actor = owner) => command(f, 'source.poll', { threadId: sub.threadId }, actor) diff --git a/core/test/project-state-runtime.test.js b/core/test/project-state-runtime.test.js index 27bde6ef..ba0e3545 100644 --- a/core/test/project-state-runtime.test.js +++ b/core/test/project-state-runtime.test.js @@ -37,9 +37,28 @@ test('original Project APIs persist Task/intake intent in SQL without model call assert.equal(disk.projects.p.goal, 'Retain the agreed product goal') assert.equal(disk.projects.p.scope, 'Original Project scope') assert.equal(modelCalls, 0) + const snapshotBody = { version: 'project-development/v1', kind: 'query', name: 'project.snapshot', projectId: 'p', payload: {} } + const baseline = (await first.rt.projectEndpoint('project-development', snapshotBody, actor)).result + const proposalBody = { version: 'project-development/v1', kind: 'command', name: 'task.propose', projectId: 'p', requestId: 'propose-through-service', + payload: { botId: 'chief', title: 'A second explicit Project change', criteria: 'Independent regression passes', impact: 'Improve the Project service' } } + const proposal = await first.rt.projectEndpoint('project-development', proposalBody, actor) + assert.equal(proposal.receipt.status, 'completed'); assert.equal(proposal.receipt.committed, true) + const proposedId = proposal.result.id + assert.equal(first.rt.state.tasks[proposedId].commandSource.commandReceiptId, proposal.receipt.id) + assert.equal((await first.rt.projectEndpoint('project-development', proposalBody, actor)).result.id, proposedId) + assert.equal(Object.keys(first.rt.state.tasks).length, 2) + const eventBody = { version: 'project-development/v1', kind: 'query', name: 'events', projectId: 'p', payload: { after: baseline.cursor } } + const facts = (await first.rt.projectEndpoint('project-development', eventBody, actor)).result + assert.deepEqual(facts.events.map(row => row.entityId), [proposedId]) + const receiptBody = { version: 'project-development/v1', kind: 'query', name: 'command.receipt', projectId: 'p', payload: { commandRequestId: proposalBody.requestId } } + assert.equal((await first.rt.projectEndpoint('project-development', receiptBody, actor)).result.id, proposal.receipt.id) await first.ctx.dispose() const second = await start() assert.equal(second.rt.stateStorage.status().backend, 'sqlite') + assert.equal((await second.rt.projectEndpoint('project-development', proposalBody, actor)).result.id, proposedId) + assert.equal(Object.keys(second.rt.state.tasks).length, 2) + assert.deepEqual((await second.rt.projectEndpoint('project-development', eventBody, actor)).result.events.map(row => row.entityId), [proposedId]) + assert.equal((await second.rt.projectEndpoint('project-development', body, actor)).result.tasks[0].taskId, task.id) assert.equal(second.rt.state.tasks[task.id].projectTask.admission.status, 'proposed') assert.equal(modelCalls, 0) diff --git a/docs/architecture.md b/docs/architecture.md index 1e0f5398..74bc6fdb 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -184,7 +184,18 @@ until explicitly migrated; a versioned guard prevents two writable authorities. History rows and immutable payload files separate storage growth from bounded model context. See [transactional storage and offline migration](subsystems/project-state-storage.md). +## Application contracts (#2185) + +The existing development service owns HTTP command/query dispatch, private durable +receipts and bounded source projections. Store commits safe event deltas alongside +business facts; the existing gateway exposes authenticated resumable SSE. Role +Handoff acceptance retains a separate durable authority proof. See +[contracts and recovery](subsystems/project-service-contracts.md). Event transport +and read projections introduce no execution workflow or extra model calls. + Project role governance (#2171) separates resident Chief proposals, Maintainer/Owner decisions, the current revisioned role catalog, and archived definitions/audit. Durable Effects proof gates visible changes; logical role keys and execution capability classes remain distinct. See [role governance](subsystems/project-agents.md#project-role-governance-2171). Native execution (#2184) uses the original Workflow lifecycle and SessionController, with exact Task-role physical bindings and durable quiescence facts. A new ephemeral carrier can restore prior context without reviving a deleted Bot identity; new input is fenced during stopping and resources are never released from Bot absence alone. See [native adapter](subsystems/project-harness.md#native-physical-session-binding-and-quiescence-2184). + +The committed service read model now covers native execution and public role-governance facts alongside SDK/Task records. Bounded entity-specific snapshot pages share a Project revision/cursor; live Task queries use committed role catalogs and existing activation/exit proofs. Exact frozen definitions remain referenced sources. diff --git a/docs/development-service-api.md b/docs/development-service-api.md index bd1c1e04..bed6e66d 100644 --- a/docs/development-service-api.md +++ b/docs/development-service-api.md @@ -547,3 +547,77 @@ until migrated. Runtime storage diagnostics/change IDs are internal ports, not a public raw-database API or authenticated event feed. See the [migration/rollback contract](subsystems/project-state-storage.md). Public bounded receipts/events are completed separately in #2185; simplified full deployment instructions remain #2189. + +## Service contract convergence (#2185) + +The envelope remains project-development/v1. Read projection versions are separate: +project-task-detail/v1, project-task-contract-summary/v1 and +project-attempt-summary/v1. Current constraints/evidence remain inline; raw lifecycle, +persona, context and Session history move behind explicit bounded source refs. +This intentionally changes unbounded raw-record assumptions. Existing UI-used fields +remain; original stored history is retained. + +| Kind/name | Payload | Result and boundary | +| --- | --- | --- | +| command task.propose | title, criteria, impact, scope?, projectTaskKind?, botId? | Proposes original Task; Chief admission remains required. Missing carrier uses existing Project Chief. Server stamps origin; no body actor/session authority. | +| query command.receipt | commandRequestId, commandName? | Calling principal's status, commit proof, origin and result ref. | +| query command.result | commandRequestId, commandName?, offset?, limit?, expectedSha256? | Exact JSON result page, max16000 chars, hash/version guarded. | +| query task.history | taskId, section, offset?, limit?, expectedSha256? | requirements, requirementsHistory, admissionDecisions, dependencies, waitHistory, results, handoffs, note, result or actors. | +| query attempt.context | attemptId, section, offset?, limit?, expectedSha256? | Prior role sections plus evidence, feedback, requirement, failure, agentDefinitions, integrationAdmissions. | +| query project.snapshot | limit?, after?, expectedRevision? | Committed bounded Project/Task snapshot and event cursor; later pages require the same revision. | +| query events | after?, limit? | Committed Project-scoped facts, max100, next/head/oldest cursors. | + +All v1 mutating envelopes have private durable receipts when the standard Effects +service is installed. Existing alias endpoints still use original domain handlers +and their own idempotency; they do not create another writable state source. Their +remaining migration/cutover is #2186. Public role acceptance/Handoff/arbitration +cannot be forged by supplying a claim or Chief role in an HTTP body. + +Receipt scope is operation + Project + authenticated user/independent agent. Reusing a key with +changed intent/preconditions is bot/idempotency-conflict. A confirmed response can +replay after its generation advances, after current permission checks. Concurrent +calls share one invocation. completion-pending-commit requires retrying the same +command, not a second execution. running/unknown does not automatically redispatch; +bot/outcome-unknown points to the receipt/target/Chief. Automatic reconciliation is +limited to existing idempotent task.run and original Task/intake proof for + task.propose/proposal.publish. Reconstructed responses say reconstructed:true. + +| Code/fact | Meaning | Client action | +| --- | --- | --- | +| bot/unauthorized, bot/forbidden | Session/principal/Project rights unavailable | Authenticate/check current rights; never borrow owner credentials. | +| bot/idempotency-conflict | Same key, changed intent | Recover the original response or submit a deliberate new command key. | +| bot/conflict | Version/source/snapshot changed | Fetch current referenced state and choose new preconditions. | +| bot/outcome-unknown | No confirmed final command outcome | Read command.receipt and target state; use existing reconciliation/Chief, not blind generic rerun. | +| bot/cursor-expired | Event epoch/retention no longer covers cursor | Fetch project.snapshot, resume after its cursor. | +| bot/busy | SSE transport capacity | Retry later or use bounded HTTP queries. | +| waitingFor:resource/worker-exit | Existing execution admission/exit boundary | Inspect queue/execution.boundary; unknown exit never authorizes another writer. | +| completion-pending-commit | Result observed, final receipt not committed | Retry the same command to finish commit without invoking again. | + +HTTP /api/bot retains {ok,value}/{ok:false,error:{code,message,details}}. Successful +v1 commands additionally return receipt, replayed and optionally reconstructed. +Normal queries and committed event generation call no model and do not change +Project positioning. Consultation remains consultation.ask/reply; do not publish a +Task for an ordinary question. + +SSE uses the existing restricted gateway: + +```sh +curl -N -H "x-flybot-member-session: $FB_SESSION" \ + "$FB_URL/api/project-events?projectId=$FB_PROJECT" +# Resume using the last accepted id from a snapshot/fact frame: +curl -N -H "x-flybot-member-session: $FB_SESSION" \ + -H "Last-Event-ID: $FB_EVENT_CURSOR" \ + "$FB_URL/api/project-events?projectId=$FB_PROJECT" +``` + +No token goes in a URL. The existing member header also accepts authenticated +personal-agent credentials. Initial frames are snapshot, then fact with id cursors; +deduplicate by cursor. access-revoked closes the stream; reset-required requires a +fresh snapshot. Initial expired cursors receive HTTP409 with nextAction. No upstream +owner cookie is forwarded. Current read rights are rechecked on each batch, including +backpressure recovery. Project sequences exclude private conversations/auth/receipt +writes. Default retention1024 facts/24h is configurable via projectEventRetention; +it is not permanent audit history. Follow Task/Handoff/Attempt source queries for +older records. See [full contract](subsystems/project-service-contracts.md). + +Operation names scope requestId, so different operations may reuse a key. When a key matches multiple operations, command.receipt/result require commandName (also supplied in resultRef); otherwise return bot/ambiguous-command. Retrying one invocation preserves its key; a deliberate new source.poll uses a fresh key. Confirmed replay preserves the original result and marks replayed:true, rather than rewriting its domain already flag. diff --git a/docs/history/2026-11-03-collaboration-repair-evidence.md b/docs/history/2026-11-03-collaboration-repair-evidence.md new file mode 100644 index 00000000..d85a62f8 --- /dev/null +++ b/docs/history/2026-11-03-collaboration-repair-evidence.md @@ -0,0 +1,57 @@ +# Earlier collaboration implementation and validation + +Preserved verbatim from PROJECT.md; active rules and current acceptance remain in active memory. + +### Collaboration repairs — 2026-11-01 + +The owner authorized sequential issues [#34–#41](https://github.com/zy95-12/FlyBot/issues/33), one PR each; see [decision](decisions/2026-11-01-collaboration-repair.md). The target is native multi-Bot reuse, per-Bot pools and enforced handoff, restored role contracts, ordinary Chief conversations, safe/watch delegation and configurable Chief inspection schedules. Public hosting is deferred; control-plane stays unchanged. + +R1 (#34) rejects incomplete repository capabilities before workers/workspaces/leases are allocated. Workbench readiness checks Git/runtime/dependency metadata and reports the precise block; this does not claim tests or SDK task delivery have run. Old failed Attempts remain readable. R1 full local verification passed in PR #42; actual shared-service integration remains pending. + +R2 (#35) restores complete Dev/Tester/Reviewer conventions, pinned policy-source hashes and final-execution prompt assembly for native and SDK roles. FlyBot code sealing requires memory/tests/architecture evidence; candidate approval explicitly cannot mint a signed forge PASS. Focused tests cover full rules reaching SDK input, required-source failures and tampering. R2 full verification and later backend/dogfood acceptance are tracked separately. + + +R3 (#36) implements per-logical-Bot pools on existing Task records, atomic claims, +versioned receipts and enforced handoff admission. New shared deliveries default +to dynamic pool dispatch; the legacy DAG is explicit opt-in. Only the receiving +role allocates an execution Session/resource. Candidate returns retain the Task +and create a new Attempt; overall completion waits for Chief judgement. Unit and +SDK contract-process tests cover non-DAG return/rework with one resource. Full +verification, conversation bridging, Chief arbitration/schedules and real shared +service dogfood remain separately tracked acceptance, not implied by these tests. + + +R4 (#37) adds conversation-only private → Chief → original-chat handoff, isolated +from private memory and other members. Query turns cannot publish or execute a +Task; scoped GitHub reads provide actual repository facts. Impactful Task kinds +persist across proposal publication; workbench status separates publication from +queue/start/execution and Chief wait. Targeted privacy, native/SDK-query, no-Task +and credential/revocation tests pass. Full verification and real shared-service +acceptance remain separate; control-plane and public hosting are unchanged. + + +R5 (#38) implements safe/private publication proposals and persistent, revocable +watch grants bound to the current human and Project. Browser logout does not +revoke watch authority; account/membership/grant changes do. Scoped actions use +current human rights and retain provenance. The unchanged SQLite platform can +supply fresh account status through a FlyBot-owned read-only operator helper, +without storing browser cookies or admin credentials. Regression tests cover +logout, restart, current rights, revocation, forged status evidence and fresh WAL +reads. Shared-service provisioning and real-model acceptance are still pending. + +R6 (#39) adds evidence-gated Chief arbitration, audited intervention and configurable +inspection on original DSH scheduled tasks. The JSON template defaults to 30 minutes; +content, cadence, thresholds and pause/enable are editable. Current configured human +authority and generation fences apply to each effect. Unknown process exits retain +capacity; a verified systemd invocation boundary permits local restart recovery. +Deterministic protocol tests return incomplete coverage to Dev then close the same +Task after a fresh Attempt. Real shared-service Codex/timer acceptance remains #41. + +Real shared-service R6 probes passed an actual Codex Chief ordinary inquiry with +zero Tasks and an original DSH 60-second wake after browser logout, then pause/ +resume and restoration to 30 minutes. R7 actual Dev requested Chief arbitration; +Chief returned the same Task to Dev with a fresh Attempt. This exposed a concurrent +resource-release reconciliation race, fixed with a red/green regression. Actual +full R7 delivery and persistent 30-minute/restart acceptance remain in progress. + +End of preserved earlier narrative. Current governing boundaries remain in PROJECT.md. diff --git a/docs/subsystems/core-host.md b/docs/subsystems/core-host.md index 4ae3252e..967fdbf4 100644 --- a/docs/subsystems/core-host.md +++ b/docs/subsystems/core-host.md @@ -972,3 +972,7 @@ A failed Attempt leaves a new Project Task blocked with its source Attempt/reaso ## Transactional persistence (#2182) The original Store supports opt-in `stateStorage: sqlite` through the [storage bridge](project-state-storage.md). It retains the same runtime objects, save chain, Project services and home lock, and acknowledges Effects after actual SQL COMMIT. Locked JSON remains the default until selected; after migration the newer-version JSON guard fences old writers and SQL remains authoritative across configuration omission. Invalid migration never creates a fresh empty team. No extra model call or periodic task is introduced. + +## Committed interfaces (#2185) + +Store builds safe per-Project event projections from each queued immutable snapshot, persists their cursor state with business records/Effects, and publishes the committed cache after rename/COMMIT. Failed saves publish nothing. Public snapshots/events omit private agent/auth/receipt data. Task construction stamps trusted command provenance once through the application context; body actor fields cannot forge it. The [contract](project-service-contracts.md) separates public requests, private receipts, host-only role authority and exact bounded history. diff --git a/docs/subsystems/install.md b/docs/subsystems/install.md index 1cb1f23f..fcff6d6f 100644 --- a/docs/subsystems/install.md +++ b/docs/subsystems/install.md @@ -92,3 +92,5 @@ entry runs `node core/src/host/workbench-gateway.js --upstream http://127.0.0.1: --upstream-cookie-file /private/mode0600-file --port 3081`; only its URL is shared with members. The private upstream file must never be committed or placed in browser assets. See [Project workbench](project-workbench.md) for the security/deployment boundary. + +#2185 retains the same restricted gateway and private upstream-cookie deployment. Optional --event-poll-ms and --event-max-streams configure its authenticated SSE transport; no token appears in an event URL. Shutdown aborts open streams. This adds no autonomous task/Chief inspection registration. Simplified standalone backend packaging remains #2189. diff --git a/docs/subsystems/project-agents.md b/docs/subsystems/project-agents.md index 0282cc9d..6c6e56d7 100644 --- a/docs/subsystems/project-agents.md +++ b/docs/subsystems/project-agents.md @@ -480,3 +480,5 @@ revocation acceptance remain in #2184; this catalog alone does not prove that lo Execution integration now preserves approved custom role keys through actual Workspaces, role-policy sources and artifact verification. Capability classes choose inherited conventions and SDK readOnly behavior; they do not grant human authority. A retired role can finish its already acquired frozen claim and explicitly hand off to an active target. Queued retired-role work becomes blocked with a Chief-routing reason; retirement never silently selects a replacement. New claims/offers to retired roles remain refused. + +Versioned service aliases `roles`, `role.proposals` and `role.decide` reuse the same catalog/proposals/Owner-Maintainer decision service. The latter retains exact-hash/configure/Chief self-approval checks and command receipt replay; it cannot grant HTTP callers Chief proposal authority. diff --git a/docs/subsystems/project-delivery.md b/docs/subsystems/project-delivery.md index 9f497b3a..8a9985bb 100644 --- a/docs/subsystems/project-delivery.md +++ b/docs/subsystems/project-delivery.md @@ -525,6 +525,8 @@ resource binding. Git regression tests exercise actual multi-parent merges and exact tree/parent sealing; their child acceptance records and publisher are explicit deterministic fixtures, not a real-provider full collaboration claim. +#2185 integration admission commands now have operation-scoped application receipts. Retries preserve the exact original admission/result with replayed:true; distinct validation probes use distinct keys and still exercise current generation, source, workspace and exit fences. The original domain admission idempotency and exact staged-tree tests remain in place. + Native Sessions now use the existing Task-role binding service through projectDelivery.nativeSessions. Canonical pool worktrees can be retained across Attempts only after exact native quiescence and released leases; original DAG workspace behavior remains compatible. Definition capability changes and removed roles block stale restart instead of inheriting authority. diff --git a/docs/subsystems/project-development-service.md b/docs/subsystems/project-development-service.md index ba1cb7c3..0f25038a 100644 --- a/docs/subsystems/project-development-service.md +++ b/docs/subsystems/project-development-service.md @@ -196,3 +196,19 @@ A failed Attempt leaves a new Project Task blocked with its source Attempt/reaso #2179 follow-up: Task views include a derived outcome reference and version for exact Chief-approved dependency replacement. This scoped metadata does not confer merge, closure or authority. Durable public receipts and bounded event/history interfaces remain #2185. + +## Contract convergence (#2185) + +The same v1 facade now supports durable principal-scoped command receipts, task.propose, bounded exact history/source reads, committed Project snapshots/events and request-scoped SSE. Task/Attempt projections retain source refs instead of retransmitting whole histories or requester Sessions. Existing handlers still own business authority and external-action reconciliation. See [command/query/event contracts](project-service-contracts.md) for scope, errors, cursor recovery, private receipt visibility and legacy compatibility limits. + +## Current role/native execution projections (#2185 composition) + +The v1 `roles` and `role.proposals` queries and `role.decide` command map to the original role-governance service (`project-roles`, `project-role-proposals`, `project-role-decision`). Decisions require current Maintainer/Owner configure authority and exact proposal hash; native Chief authority is still required to propose. A generic HTTP caller cannot impersonate Chief or self-approve. Command receipts deduplicate Owner decisions; personal-agent safe/watch boundaries remain. + +Task views distinguish current `roleDefinitionRevision` from pinned `execution.definitionRevision`, logical/custom role identity from its `policyRole`, activation intent from an actually active turn, stopping from unknown stop, and confirmed exit from Bot absence. Live activation checks use the existing Effects acknowledgement. Execution metadata exposes only scoped identifiers/status/proof kind, never process-scope details, credentials or raw host failures. A physically released execution may still wait for the existing handoff decision; exit does not close a Task. Current catalogs come through the governance committed-view port, preserving pending-change visibility rules. + +Attempt lists retain at most32 role Session slots and three historical Sessions with bounded metadata/source references. Frozen definitions/Skills and exact context stay available through hash-paged `attempt.context`; they are not copied into every list result. Historical definition references point to `roleSessionHistory`, not the current slot. + +Committed Project events include role catalog revisions, public role-governance proposal statuses and native execution activation/exit facts. Private personal-agent proposals/conversations remain excluded. `project.snapshot` retains its default Task page; callers may select `payload.entityType` as `task`, `attempt`, `execution`, `resource`, `input` or `role-proposal` to rebuild other public read-model pages. Non-Task pages return `entities`; all types share the Project revision/event cursor and existing1–100 limit. Continuing pages requires the same `expectedRevision`; unknown/private entity types are rejected. After expiry fetch fresh snapshot pages and resume from that Project cursor. Receiving any snapshot/event grants no execution or approval authority. + +Legacy Projects with an implicit built-in catalog report its effective revision1 without materializing it on read. Queries use the raw committed governance catalog, not full Skill/memory execution-context resolution. Explicit configuration/proposals persist through the existing write path. A Project/role read must not create an otherwise unrelated Project change event on the next command. diff --git a/docs/subsystems/project-effects.md b/docs/subsystems/project-effects.md index 784c12d1..c807cb5c 100644 --- a/docs/subsystems/project-effects.md +++ b/docs/subsystems/project-effects.md @@ -143,3 +143,5 @@ generation, independently of a newer unsaved receipt revision. It cannot authori an uncommitted replacement, unknown protocol or absent token. Delivery requires this proof before using pins; a successful/no-op save alone is not enough. This metadata-only adapter does not launch code or imply a merge/verification result. + +#2185 adds pure pool.accept and service.command commit markers. They acknowledge ownership/command phases without invoking another executor. service.command results bind their exact SHA256; marker list/retry visibility follows the receipt's original independent principal, including native/external personal-agent identity across carrier rotation. Other Project members do not gain access to private command receipts or their markers. diff --git a/docs/subsystems/project-pools.md b/docs/subsystems/project-pools.md index 2af8f9de..c4efaa0d 100644 --- a/docs/subsystems/project-pools.md +++ b/docs/subsystems/project-pools.md @@ -171,6 +171,8 @@ Project Tasks may split independent children after actual source exit, preservin #2179 follow-up: After adjudicated child results, continuation reuses the original logical role. An unsealed Attempt remains; a sealed candidate requires one successor Attempt seeded from that candidate. Replay reuses the recorded successor and stable children-wait Handoff. Claiming records the wait as resumed, preserving its original released claim reference. +#2185 acceptance: claim/accept persists a pool.accept marker alongside the claim and Handoff acceptance. Failed commits restore pending ownership; idempotent replay/current role operations require confirmed acceptance proof. The adapter is registered on startup for marker recovery, not only after a new claim. Legacy persisted claims without the additive proof retain their original recovery fence; the marker never dispatches an executor. + #2184 records logical role and pinned capability class on a claim. Active targets alone can receive new offers/claims; a removed queued target becomes blocked with a Chief-routing reason while its Task/initiative/history remain intact. An already acquired source may finish its retained frozen claim after retirement, then explicitly hand off to an active target. Developer-capability destinations use existing rework semantics even when named differently. Artifact responsibilities stay in the existing Attempt's roleAssignments; no new queue or scheduler is added. Composition with #2184 rechecks the committed target role after asynchronous source-exit confirmation, before claiming. Retirement during inspection leaves the original Task blocked for Chief routing; it cannot acquire a new execution claim from the earlier catalog snapshot. diff --git a/docs/subsystems/project-service-contracts.md b/docs/subsystems/project-service-contracts.md new file mode 100644 index 00000000..e1885ea6 --- /dev/null +++ b/docs/subsystems/project-service-contracts.md @@ -0,0 +1,126 @@ +--- +subsystem: project-service-contracts +owned_paths: + - core/src/host/project-command-receipts.js + - core/src/host/project-service-events.js + - core/src/host/project-service-views.js + - core/src/host/project-service-sse.js + - core/test/project-command-receipts.test.js + - core/test/project-service-events.test.js + - core/test/project-service-sse.test.js +--- + +# Commands, committed views and event transport + +The existing `project-development/v1` application boundary and `/api/bot` remain. +There is no second Project, Task, identity, queue or executor. Public HTTP never +accepts a role/Chief execution token; the existing trusted role ports own Handoff +and adjudication. Consultation remains a separate message contract, not a Task. + +## Durable public commands + +The v1 command envelope requires a stable requestId. Receipts are scoped by operation, Project, +user and independent personal-agent identity. Native watch and external connections +for the same independent agent can recover the same receipt across carrier changes; +legacy delegated carriers keep grant isolation. Origin is server-derived, never a +body actor/session field. Human, personal-agent and Project-Chief origins differ. + +Canonical command intent includes operation, payload and supplied preconditions. +Omitted candidate conditions differ from explicitly requiring no candidate. Changing +intent under the same key returns `bot/idempotency-conflict`. A confirmed replay +rechecks current authority and returns its recorded response even if the original +generation has advanced. First dispatch checks current target/generation both before +and after committing its begin marker. Concurrent requests share one invocation. + +Existing Effects commit intent, begin and result markers with receipt state. Result +proof binds its SHA256. Failed intent cannot dispatch. A known result awaiting its +final commit reports `completion-pending-commit`; retry finishes that commit without +repeating the handler. Results above64KiB use immutable private hashed files, bounded +to8MiB; exact reads reject symlinks, hash changes and foreign paths. `command.result` +returns at most16000 characters with a hash guard. Result metadata/history remains +outside default model context. Receipts are retained for recovery; automatic archival +or garbage collection is not introduced here. + +A durable begin without a confirmed result is **unknown**, not permission to blindly +repeat. Explicit reconciliation is limited to existing idempotent `task.run` and +`task.propose`/`proposal.publish` recovery from their server-stamped original Task and +committed intake proof. Reconstructed responses are labelled. Other ambiguous +commands return `bot/outcome-unknown` with a receipt/target/Chief query suggestion. +This is not a universal exactly-once external-side-effect guarantee. Existing domain +adapters still reconcile their own external actions. + +`command.receipt` and `command.result` are private to the calling principal. Private +command marker Effects are also excluded from other principals' list/retry views. +Role acceptance separately commits `pool.accept`: failed saves restore the pending +Handoff, retries cannot return an uncommitted claim, and current role operations +require its committed acceptance proof. No new role execution is launched by a +receipt marker or by receiving an event. + +## Bounded queries + +Task/Attempt read projections have explicit projectionVersion/preview markers. +They retain current constraints, phase, provenance and evidence metadata, not entire +requirements/wait/result/session histories. No execution-requester Session appears +in Task summaries. Existing UI-consumed fields remain; unbounded raw-record clients +must follow source references rather than depend on removed historical payloads. +Original stored records and logical/physical execution Sessions are not truncated. + +- `task.history`: requirements, requirement versions, admission decisions, dependency + results, wait history, final results, handoffs, notes and actor history. +- `attempt.context`: existing role contexts plus exact evidence, feedback, + requirement, failure, role definitions and integration-admission sources. +- Exact sources are Project/Task scoped, max16000 characters, with SHA256 and + expectedSha256/offset guards. Unknown sections/foreign targets are refused. +- Attempt pages remain max50. Repeated role Session previews show the latest three, + retaining references to complete original sources. Persona, full logs and tool + transcripts do not ride on every Task detail response. + +## Committed event model + +The original Store prepares safe Project/Task/Attempt/execution/resource/input +projections inside its serialized immutable-snapshot write. It persists changed +facts and cursor metadata in the same JSON rename or SQLite transaction as business +state/Effects. Only after that commit does it acknowledge the event read model. +A failed save cannot publish an event or a newer committed snapshot. + +Each Project has its own epoch, sequence and revision. Private chat, unpublished +proposals, personal memory, credentials and command-receipt writes do not perturb +its public sequence. Event data includes bounded public metadata/ref IDs; no raw +operator configuration, auth Session, tool transcript or private conversation. +Permissions are checked on every query and transport batch. Events describe stored +facts, not proof of live process activity or authority to execute a command. + +`project.snapshot` returns a bounded committed Project/Task projection with cursor +and revision. Further Task pages require expectedRevision; changed snapshots must +restart. `events` reads after a Project-scoped cursor, max100. Defaults retain1024 +facts per Project and24hours, configurable through `projectEventRetention` with +maxPerProject and maxAgeMs. Clock rollback cannot create a hole; duplicate/gapped +persisted streams fail closed. Age/size expiry or a different epoch returns +`bot/cursor-expired` and `nextAction: project.snapshot`. SSE is not permanent history; +Task/Handoff/Attempt queries retain the original history. Delivery is at least once; +clients deduplicate by cursor and query current state when uncertain. + +## HTTP/SSE + +`GET /api/project-events?projectId=PROJECT_ID` on the existing restricted gateway +requires `x-flybot-member-session`, accepting the existing user or personal-agent +credential. Tokens in URLs are refused. Initial connection emits a bounded snapshot; +Last-Event-ID (or an unambiguous after query parameter) resumes committed facts. +No DSH owner cookie, Set-Cookie or raw DSH event stream is forwarded to the client. + +SSE uses authenticated original application queries. Connection-bound polling is +transport only, not a Chief inspection/work timer. Factory options eventPollMs +(default1000) and eventMaxStreams(default64) bound it; the gateway CLI exposes +--event-poll-ms and --event-max-streams. Backpressure reauthenticates before another +batch, revocation closes the stream, and server shutdown/disconnect aborts polling. +Initial errors use HTTP401/403/409/429 as appropriate. An established stream sends +access-revoked or reset-required and closes; clients must act rather than reconnect +forever with an expired cursor. + +Tests distinguish actual TCP/SQLite/process operations from fixture authorization +and deterministic inference. This stage does not claim an external grok/dots provider +or full real-SDK parallel development acceptance. Compatibility cutover is #2186; +simplified real backend deployment/README remains #2189. No control-plane/UI rewrite, +new autonomous worker, inspection timer, deployment or merge is performed. + +Operation names scope requestId, so different operations may reuse a key. When a key matches multiple operations, command.receipt/result require commandName (also supplied in resultRef); otherwise return bot/ambiguous-command. Retrying one invocation preserves its key; a deliberate new source.poll uses a fresh key. Confirmed replay preserves the original result and marks replayed:true, rather than rewriting its domain already flag. diff --git a/docs/subsystems/project-sources.md b/docs/subsystems/project-sources.md index 56844bdc..51b509f8 100644 --- a/docs/subsystems/project-sources.md +++ b/docs/subsystems/project-sources.md @@ -105,3 +105,5 @@ Tasks, exact-SHA history, lossless chunks and the native schedule adapter. Mock observations are separate from the actual UID998 scoped GitHub REST read. A real GitHub feedback→current frontend Bot→handoff acceptance must be recorded separately; a successful read or source test is not proof of that complete path. + +#2185 keeps domain rollback/revocation tests on the original facade aliases so an injected source-transaction failure is not consumed by an earlier generic command-intent save. Actual v1 subscription/polling with the real Store remains covered separately. A repeated HTTP command key replays its prior response; a new observation uses a fresh key, while the Source engine still deduplicates external revisions and input IDs. diff --git a/docs/subsystems/project-workbench.md b/docs/subsystems/project-workbench.md index d8af5901..33e1cce6 100644 --- a/docs/subsystems/project-workbench.md +++ b/docs/subsystems/project-workbench.md @@ -473,5 +473,7 @@ Project execution quota excludes a parent waiting for independently admitted chi The legacy Workflow gateway now also excludes Chief control jobs from execution quota. A real DSH admission probe reproduced a first delivery failing while its Chief intake was still finishing; the shared quota regression protects both the gateway and scheduler. +#2185 exposes request-scoped GET /api/project-events on the existing gateway. It forwards only authenticated Project service queries, requires credentials in headers, supports Last-Event-ID, bounds connections/backpressure and closes on revoke/shutdown. It does not alter workbench UI components, forward the private owner cookie, or create a parallel state store. Default1000ms transport polling is configurable and unrelated to Chief inspection cadence. + #2171 adds role catalog/proposal/decision endpoints to the existing restricted gateway allowlist. Reads preserve Project scope; decisions keep configure authorization in the domain service. No control-plane or frontend replacement is introduced. diff --git a/patches/ds-bot/0002-flybot-api-host.patch b/patches/ds-bot/0002-flybot-api-host.patch index 0279e258..1b1b083c 100644 --- a/patches/ds-bot/0002-flybot-api-host.patch +++ b/patches/ds-bot/0002-flybot-api-host.patch @@ -699,13 +699,14 @@ diff --git a/src/host/roster.js b/src/host/roster.js diff --git a/src/host/store.js b/src/host/store.js --- a/src/host/store.js +++ b/src/host/store.js -@@ -1,21 +1,51 @@ +@@ -1,21 +1,53 @@ // Store: the saved team (state.json), its migrations, the lock that keeps a second dsh // host on the same data from writing it, and the indexes that map a Session to the Bot // or group chat it belongs to. -import { mkdirSync, statSync } from 'node:fs' +import { install as installProjects } from './project-service.js' +import { createProjectStateStorage } from './project-state-storage.js' ++import { createProjectServiceEvents } from './project-service-events.js' +import { constants, mkdirSync, statSync } from 'node:fs' import { copyFile, mkdir, readFile, rename, unlink, writeFile } from 'node:fs/promises' import { join } from 'node:path' @@ -744,6 +745,7 @@ diff --git a/src/host/store.js b/src/host/store.js + // Project services own domain records; store owns their durable snapshot. + // projectRuns is a legacy projection; projectExecutions holds direct Run ownership. + const state = { version: STATE_VERSION, mainBotIds: [], members: {}, bots: {}, rooms: {}, agents: {}, capabilities: { items: {} }, exchanges: [], exchangeTurns: {}, questions: {}, prefs: {}, invitations: {}, authSessions: {}, tasks: {}, retiredSessions: {}, projects: {}, projectRuns: {}, projectMemory: {}, revision: 0 } ++ const committedEvents = createProjectServiceEvents(config.projectEventRetention) let loaded = null let writing = Promise.resolve() + // Every save this host has accepted since it started — a monotonic watermark, never decremented @@ -966,7 +968,7 @@ diff --git a/src/host/store.js b/src/host/store.js // LEGACY: Bots saved with a lab ran on that lab's default model; the model they // last ran on becomes their own, so dropping the lab changes nothing for them. if ('lab' in bot) { -@@ -161,18 +161,101 @@ export function install(rt) { +@@ -161,18 +161,102 @@ export function install(rt) { migrated = true } // Unnamed groups from before names followed their members may still list old names. @@ -1027,6 +1029,7 @@ diff --git a/src/host/store.js b/src/host/store.js + if (rt.projectEffects?.recover()) migrated = true + if (syncDesignatedChief()) migrated = true indexSessions() ++ committedEvents.load(state) return { migrated: migrated || stale.length > 0, stale } } @@ -1154,22 +1157,27 @@ diff --git a/src/host/store.js b/src/host/store.js const snapshot = JSON.stringify(state, null, 2) const run = writing.then(async () => { const temp = `${statePath}.${process.pid}.tmp` -@@ -256,13 +256,18 @@ export function install(rt) { +@@ -256,13 +256,23 @@ export function install(rt) { } } ensureLock() - await writeFile(temp, snapshot, 'utf8') - ensureLock() - await rename(temp, statePath) -+ if (storage.sqlite) storage.write(snapshot) ++ const prepared = committedEvents.prepare(JSON.parse(snapshot)) ++ const persistedSnapshot = JSON.stringify(prepared.snapshot, null, 2) ++ if (storage.sqlite) storage.write(persistedSnapshot) + else { -+ await writeFile(temp, snapshot, 'utf8') ++ await writeFile(temp, persistedSnapshot, 'utf8') + ensureLock() + await rename(temp, statePath) + } + rt.projectEffects?.saved(effectSnapshot) ++ committedEvents.saved(prepared) ++ state.projectServiceEventStreams = structuredClone(prepared.snapshot.projectServiceEventStreams) ++ state.projectServiceEvents = structuredClone(prepared.snapshot.projectServiceEvents) seen = stamp() -+ await journal?.record(snapshot) ++ await journal?.record(persistedSnapshot) rt.capabilitiesStateSaved?.() } catch (error) { - await unlink(temp).catch(() => {}) @@ -1241,7 +1249,7 @@ diff --git a/src/host/store.js b/src/host/store.js rt.cancelApprovals?.(sessionId) // A plain archive refuses a running Session, which would go on working for a group // that no longer has it. -@@ -335,23 +335,29 @@ export function install(rt) { +@@ -335,23 +335,30 @@ export function install(rt) { } const findBot = (ref) => { if (ref === undefined) return undefined @@ -1267,6 +1275,7 @@ diff --git a/src/host/store.js b/src/host/store.js Object.assign(rt, { - state, load, save, refresh, readOnly, + state, load, save, refresh, readOnly, mutate, savesQueued, savesSettled, ++ committedProjectEvents: { query: (id, options) => committedEvents.query(id, options), snapshot: (id, options) => committedEvents.snapshot(id, options) }, + stateStorage: { status: () => storage.status(), changes: options => storage.changes(options) }, botOf, roomOf, isMain, chatOf, selfOf, isBotSession, chatOwners, groupOwners, pastOwners, diff --git a/patches/ds-bot/files/src/host/project-agents.js b/patches/ds-bot/files/src/host/project-agents.js index 193667be..d47045ef 100644 --- a/patches/ds-bot/files/src/host/project-agents.js +++ b/patches/ds-bot/files/src/host/project-agents.js @@ -109,8 +109,8 @@ export function createProjectAgents(rt) { rt.requireProjectAccess(authority.actor, authority.agent.projectId) return authority } - function storedDefinitionSet(projectId) { - const existing = ownEntry(definitions(), projectId) + function storedDefinitionSet(projectId, materialize = true) { + const existing = ownEntry(state.projectAgentDefinitions ?? {}, projectId) if (existing) return existing const model = rt.config.projectModel ?? rt.cachedDefault?.() ?? '' const workerNodes = deliveryGraph(model).nodes @@ -118,10 +118,10 @@ export function createProjectAgents(rt) { ...workerNodes.map(node => ({ role: node.id, name: node.name, instructions: node.prompt, model: node.model, skills: node.id === 'developer' ? ['code-change'] : node.id === 'tester' ? ['test-evidence'] : ['commit-review'], tools: [] }))] const record = { projectId, revision: 1, roles: rows.map(row => ({ ...row, id: `${projectId}:${row.role}`, memoryRefs: [], execution: { harness: 'flybot', resourceIds: [] }, toolPolicy: row.role === 'chief' ? { kind: 'scoped-project', filesystem: false, connectors: [] } : { kind: 'registered-worktree', shell: true, connectors: [], isolation: 'cwd-only' } })) } - setEntry(definitions(), projectId, record) + if (materialize) setEntry(definitions(), projectId, record) return record } - function definitionSet(projectId) { return roleGovernance ? roleGovernance.visibleSet(projectId, storedDefinitionSet(projectId)) : storedDefinitionSet(projectId) } + function definitionSet(projectId) { return roleGovernance ? roleGovernance.visibleSet(projectId, storedDefinitionSet(projectId, false)) : storedDefinitionSet(projectId, false) } async function prepareDefinition(role, body, projectId) { if (body.tools !== undefined && !Array.isArray(body.tools)) throw refuse('Invalid role tools') if (Array.isArray(body.tools) && body.tools.length) throw refuse('Agent connector tools need explicitly delegated credentials') @@ -150,7 +150,7 @@ export function createProjectAgents(rt) { rt.projectResources?.checkPool(projectId, next.execution) return next } - roleGovernance = createProjectRoleGovernance(rt, { definitionSet: storedDefinitionSet, prepareDefinition, + roleGovernance = createProjectRoleGovernance(rt, { definitionSet: storedDefinitionSet, readDefinitionSet: projectId => storedDefinitionSet(projectId, false), prepareDefinition, baseDefinition(projectId, role) { const active = definitionSet(projectId).roles.find(row => row.role === role) if (active) return structuredClone(active) @@ -196,6 +196,8 @@ export function createProjectAgents(rt) { return known } if (project.status !== 'active') throw refuse('Project is archived; a new agent cannot be created') + // Creating an actual carrier is a write; metadata reads only peek defaults. + storedDefinitionSet(projectId) let bot if (visibility === 'public' && !room) { const previous = project.chief diff --git a/patches/ds-bot/files/src/host/project-command-receipts.js b/patches/ds-bot/files/src/host/project-command-receipts.js new file mode 100644 index 00000000..07bb9843 --- /dev/null +++ b/patches/ds-bot/files/src/host/project-command-receipts.js @@ -0,0 +1,165 @@ +// Public command recovery over original handlers. Unknown execution outcomes are +// retained, never blindly redispatched. Role acceptance has its own scoped proof. +import { AsyncLocalStorage } from 'node:async_hooks' +import { createHash } from 'node:crypto' +import { constants, closeSync, existsSync, fstatSync, lstatSync, openSync, readSync } from 'node:fs' +import { join } from 'node:path' +import { writeDurableStateFile } from './project-state-storage.js' +import { sourcePage } from './project-service-views.js' +const hash = value => createHash('sha256').update(typeof value === 'string' ? value : JSON.stringify(value)).digest('hex') +const stable = value => Array.isArray(value) ? value.map(stable) : value && typeof value === 'object' ? Object.fromEntries(Object.keys(value).sort().map(key => [key, stable(value[key])])) : value +const fail = (message, code = 'bot/conflict', details = {}) => Object.assign(new Error(message), { code, details }) +export function commandPrincipal(actor, state) { + const carrier = actor.watchGrantId && state?.projectAgents?.[actor.agentId] + const agentId = actor.personalAgentCredentialId ? actor.agentId : carrier?.personalAgentId ?? actor.agentId ?? null + return hash([actor.memberId, agentId, actor.watchGrantId && !carrier?.personalAgentId ? actor.watchGrantId : null]) +} +export function commandEffectVisible(state, effect, actor) { + const id = effect.payload?.commandReceiptId + if (!id) return true + const receipt = Object.hasOwn(state.projectCommandReceipts ?? {}, id) ? state.projectCommandReceipts[id] : null + return Boolean(receipt && receipt.projectId === effect.projectId && receipt.principal === commandPrincipal(actor, state)) +} +export function createCommandReceipts(rt) { + const pending = new Map(), context = new AsyncLocalStorage() + const table = () => (rt.state.projectCommandReceipts ??= {}) + const identity = (projectId, name, requestId, actor) => `command-${hash([projectId, commandPrincipal(actor, rt.state), name, requestId])}` + const rows = () => rt.state.projectCommandReceipts ?? {} + const lookup = id => Object.hasOwn(rows(), id) ? rows()[id] : undefined + function resultCommitted(row) { + const effect = row.resultEffect && rt.state.projectEffects?.[row.resultEffect.id] + return Boolean(rt.projectEffects.intentCommitted(row.resultEffect) && effect?.payload.commandReceiptId === row.id && effect.payload.phase === 'result' && effect.payload.resultSha256 === row.resultSha256) + } + const summary = row => { + const committed = row.status === 'completed' ? resultCommitted(row) : rt.projectEffects.intentCommitted(row.beginEffect ?? row.intentEffect) + const status = row.status === 'completed' && !committed ? 'completion-pending-commit' : row.status + return { version: row.version, id: row.id, projectId: row.projectId, requestId: row.requestId, name: row.name, status, committed, origin: structuredClone(row.origin), createdAt: row.createdAt, updatedAt: row.updatedAt, + resultSha256: row.resultSha256 ?? null, resultBytes: row.resultBytes ?? null, resultRef: row.resultSha256 ? { query: 'command.result', commandName: row.name, commandRequestId: row.requestId, expectedSha256: row.resultSha256 } : null, + outcome: ['running', 'unknown'].includes(status) ? 'unknown-until-reconciled' : status, nextAction: ['running', 'unknown'].includes(status) ? 'query-target-or-Chief' : status === 'completion-pending-commit' ? 'retry-command' : null } + } + rt.projectEffects.register('service.command', { run: effect => ({ commandReceiptId: effect.payload.commandReceiptId, phase: effect.payload.phase, status: 'durable-command-marker' }) }) + function commandOrigin(spec) { + const actor = spec.actor, carrier = actor.watchGrantId && rt.state.projectAgents?.[actor.agentId] + const agentId = actor.personalAgentCredentialId ? actor.agentId : carrier?.personalAgentId + return { kind: actor.chiefJobId ? 'project-chief' : agentId ? 'personal-agent' : actor.watchGrantId || actor.agentId ? 'delegated-bot' : 'human', memberId: actor.memberId, + ...(agentId ? { agentId } : {}), ...(actor.chiefJobId ? { botId: `${spec.projectId}:chief` } : {}) } + } + async function authorize(spec) { + if (rt.state.members?.[spec.actor.memberId]?.status === 'disabled') throw fail('Command principal was disabled', 'bot/forbidden') + if (rt.validateHumanActor && !await rt.validateHumanActor(spec.actor)) throw fail('Command principal was revoked', 'bot/forbidden') + if (!rt.platformIdentityEnabled && !spec.actor.agentId && !spec.actor.watchGrantId && !spec.actor.chiefJobId && spec.actor.sessionId) { + const session = rt.state.authSessions?.[spec.actor.sessionId] + if (!session || session.memberId !== spec.actor.memberId || session.revokedAt) throw fail('Command session was revoked', 'bot/unauthorized') + } + const task = spec.taskId && rt.state.tasks?.[spec.taskId] + rt.requireProjectAccess(spec.actor, spec.projectId, spec.action ?? 'read', task) + } + async function commit(row, phase) { + const token = rt.projectEffects.stage({ kind: 'service.command', key: `${row.id}:${phase}`, projectId: row.projectId, + payload: { commandReceiptId: row.id, phase, intentSha256: row.intentSha256, ...(phase === 'result' ? { resultSha256: row.resultSha256 } : {}) } }) + row[`${phase}Effect`] = token + try { await rt.projectEffects.persist(token) } + catch (error) { rt.projectEffects.discardUncommitted(token); throw error } + } + function resultOf(row) { + let text = row.resultText + if (row.resultFile) { + if (!/^[a-f0-9]{64}$/.test(row.resultSha256)) throw fail('Invalid command result reference') + const file = join(rt.home, 'project-command-results', row.resultSha256 + '.json') + if (!existsSync(file)) throw fail('Command result is unavailable', 'bot/outcome-unknown', { nextAction: 'query-target-or-Chief' }) + const info = lstatSync(file) + if (!info.isFile() || info.isSymbolicLink() || info.size > 8 * 1024 * 1024) throw fail('Command result must be a bounded regular file', 'bot/outcome-unknown') + const fd = openSync(file, constants.O_RDONLY | (constants.O_NOFOLLOW ?? 0)) + try { + const current = fstatSync(fd) + if (current.ino !== info.ino || current.dev !== info.dev || current.size !== info.size) throw fail('Command result file changed', 'bot/outcome-unknown') + const buffer = Buffer.alloc(info.size + 1), size = readSync(fd, buffer, 0, buffer.length, 0) + if (size !== info.size) throw fail('Command result file changed during read', 'bot/outcome-unknown') + text = buffer.subarray(0, size).toString('utf8') + } finally { closeSync(fd) } + } + if (typeof text !== 'string' || Buffer.byteLength(text) > 8 * 1024 * 1024 || hash(text) !== row.resultSha256) throw fail('Command result hash changed', 'bot/outcome-unknown', { nextAction: 'query-target-or-Chief' }) + return JSON.parse(text) + } + async function complete(row, result) { + const text = JSON.stringify(result ?? null), bytes = Buffer.byteLength(text) + if (bytes > 8 * 1024 * 1024) throw fail('Command result exceeds its recovery bound', 'bot/outcome-unknown', { nextAction: 'query-target-or-Chief' }) + row.resultSha256 = hash(text); row.resultBytes = bytes + if (bytes > 65536) { writeDurableStateFile(join(rt.home, 'project-command-results', row.resultSha256 + '.json'), text, { exclusive: true }); row.resultFile = true } + else row.resultText = text + row.status = 'completed'; row.updatedAt = Date.now() + await commit(row, 'result') + } + async function run(spec, invoke, check) { + await authorize(spec) + const id = identity(spec.projectId, spec.name, spec.requestId, spec.actor), intentSha256 = hash(stable({ name: spec.name, args: spec.args, expectedGeneration: spec.expectedGeneration, expectedCandidateSha: spec.expectedCandidateSha })) + const prior = lookup(id) + if (prior && (prior.version !== 'project-command/v1' || prior.intentSha256 !== intentSha256)) throw fail('Command requestId was reused with different content', 'bot/idempotency-conflict') + if (pending.has(id)) { const result = await pending.get(id); await authorize(spec); return result } + const operation = (async () => { + let row = prior + if (row?.status === 'completed') { + if (!resultCommitted(row)) await commit(row, 'result') + await authorize(spec) + return { result: resultOf(row), receipt: summary(row), replayed: true } + } + if (row && ['running', 'unknown'].includes(row.status)) { + const recovered = await spec.reconcile?.(row) + if (!recovered) throw fail('Previous command outcome needs reconciliation; it was not re-executed', 'bot/outcome-unknown', { receiptId: row.id, nextAction: 'command.receipt', targetAction: 'query-target-or-Chief' }) + await complete(row, recovered.result) + await authorize(spec) + return { result: resultOf(row), receipt: summary(row), replayed: true, reconstructed: true } + } + if (row?.status === 'rejected') throw fail('Command was rejected; use a new requestId for changed preconditions', row.failure?.code ?? 'bot/conflict', { nextAction: 'new-command' }) + check?.() + if (!row) { + row = { id, version: 'project-command/v1', projectId: spec.projectId, requestId: spec.requestId, name: spec.name, principal: commandPrincipal(spec.actor, rt.state), intentSha256, + origin: commandOrigin(spec), status: 'prepared', createdAt: Date.now(), updatedAt: Date.now() } + Object.defineProperty(table(), id, { value: row, enumerable: true, writable: true, configurable: true }) + try { await commit(row, 'intent') } + catch (error) { if (!rt.projectEffects.intentCommitted(row.intentEffect) && lookup(id) === row) delete table()[id]; throw error } + } + if (!rt.projectEffects.intentCommitted(row.intentEffect)) await commit(row, 'intent') + row.status = 'running'; row.updatedAt = Date.now() + try { await commit(row, 'begin') } + catch (error) { if (!rt.projectEffects.intentCommitted(row.beginEffect)) row.status = 'prepared'; throw error } + let invoked = false + try { + await authorize(spec); check?.(); invoked = true + const result = await context.run({ commandReceiptId: id, projectId: spec.projectId, name: spec.name, origin: row.origin, taskStamped: false }, invoke) + await complete(row, result) + await authorize(spec) + return { result: resultOf(row), receipt: summary(row), replayed: false } + } catch (error) { + if (row.status !== 'completed') { + row.status = invoked ? 'unknown' : 'rejected'; row.updatedAt = Date.now() + row.failure = { code: typeof error.code === 'string' && error.code.startsWith('bot/') ? error.code : 'bot/command-failed' } + // Retain ambiguous outcome even if recording it fails. The durable begin + // marker still fences automatic replay after a restart. + try { await commit(row, 'failure') } catch { /* Existing begin intent remains the conservative recovery fact. */ } + } + throw error + } + })() + pending.set(id, operation) + try { return await operation } finally { if (pending.get(id) === operation) pending.delete(id) } + } + function get(projectId, requestId, actor, commandName) { + rt.requireProjectAccess(actor, projectId) + if (typeof requestId !== 'string' || !/^[A-Za-z0-9_-]{1,100}$/.test(requestId)) throw fail('Choose a stable command requestId', 'bot/invalid') + const matches = commandName === undefined ? Object.values(rows()).filter(row => row.projectId === projectId && row.principal === commandPrincipal(actor, rt.state) && row.requestId === requestId) : [lookup(identity(projectId, commandName, requestId, actor))].filter(Boolean) + if (matches.length > 1) throw fail('Choose commandName for this reused operation-scoped requestId', 'bot/ambiguous-command', { nextAction: 'specify-command-name' }) + const row = matches[0] + if (!row) throw fail('Command receipt is unavailable to this principal', 'bot/not-found') + return row + } + return { run, source(projectId) { + const current = context.getStore() + if (!current || current.projectId !== projectId || !['task.propose', 'proposal.publish'].includes(current.name) || current.taskStamped) return null + current.taskStamped = true + return { commandReceiptId: current.commandReceiptId, projectId, origin: structuredClone(current.origin) } + }, + get: (projectId, requestId, actor, commandName) => summary(get(projectId, requestId, actor, commandName)), + result(projectId, body, actor) { const row = get(projectId, body.commandRequestId, actor, body.commandName); if (row.status !== 'completed' || !resultCommitted(row)) throw fail('Command result is not committed', 'bot/outcome-unknown'); return { receiptId: row.id, ...sourcePage(resultOf(row), body, hash) } }, + } +} diff --git a/patches/ds-bot/files/src/host/project-development-service.js b/patches/ds-bot/files/src/host/project-development-service.js index e5c04918..9a8c29ca 100644 --- a/patches/ds-bot/files/src/host/project-development-service.js +++ b/patches/ds-bot/files/src/host/project-development-service.js @@ -1,5 +1,7 @@ // One application boundary over existing Project services; no second executor/store. import { ownEntry } from './store.js' +import { createCommandReceipts } from './project-command-receipts.js' +import { executionMetadataView, taskContractView, taskProvenance, taskDetailView, attemptMetadataView, taskHistorySource, sourcePage } from './project-service-views.js' import { createHash } from 'node:crypto' import { taskOutcome } from './project-task-lifecycle.js' import { inspectionFacts } from './project-chief-control.js' @@ -13,8 +15,9 @@ const sameTask = (row, task) => Boolean(row && row.projectId === task.projectId const roles = new Set(['chief', 'developer', 'tester', 'reviewer']) const operations = Object.freeze({ 'consultation.reply': 'project-consultation-reply', 'consultation.ask': 'project-consultation-ask', 'consultation.get': 'project-consultation-get', 'consultation.list': 'project-consultation-list', + 'roles': 'project-roles', 'role.proposals': 'project-role-proposals', 'role.decide': 'project-role-decision', 'handoff.get': 'project-handoff-get', 'handoff.list': 'project-handoff-list', - 'task.change.request': 'project-task-change-request', 'task.run': 'project-task-run', 'task.detail': 'project-task-detail', + 'task.propose': 'task-create', 'task.change.request': 'project-task-change-request', 'task.run': 'project-task-run', 'task.detail': 'project-task-detail', 'task.accept': 'project-task-accept', 'task.reject': 'project-task-reject', 'attempt.stop': 'project-attempt-stop', 'attempt.cleanup': 'project-attempt-cleanup', 'attempt.integration-state': 'project-attempt-integration-state', 'attempt.integration-bind': 'project-attempt-integration-bind', @@ -28,12 +31,12 @@ const operations = Object.freeze({ 'execution.boundary': 'project-execution-boundary', 'pools': 'project-task-pools', 'effect.retry': 'project-effect-retry', 'comment.publish': 'project-github-comment', }) -const queries = new Set(['consultation.get', 'consultation.list', 'handoff.get', 'handoff.list', 'tasks', 'task', 'task.detail', 'inputs', 'sources', 'execution.boundary', 'effects', 'attempt.context', 'attempt.integration-state', 'attempt.log', 'chief.state', 'pools']) +const queries = new Set(['roles', 'role.proposals', 'consultation.get', 'consultation.list', 'handoff.get', 'handoff.list', 'task.history', 'project.snapshot', 'events', 'command.receipt', 'command.result', 'tasks', 'task', 'task.detail', 'inputs', 'sources', 'execution.boundary', 'effects', 'attempt.context', 'attempt.integration-state', 'attempt.log', 'chief.state', 'pools']) const legacy = new Set(Object.values(operations)) const readEndpoints = new Set([...queries].map(name => operations[name]).filter(Boolean)) export const developmentQuery = body => record(body) && body.version === DEVELOPMENT_PROTOCOL && body.kind === 'query' && queries.has(body.name) -export function taskView(state, task, { activity = {} } = {}) { +export function taskView(state, task, { activity = {}, roleCatalog, isIntentCommitted = () => false } = {}) { const c = task.collaboration const candidate = ownEntry(state.projectAttempts ?? {}, task.deliveryAttemptId) const attempt = sameTask(candidate, task) ? candidate : null @@ -41,15 +44,21 @@ export function taskView(state, task, { activity = {} } = {}) { const claim = c?.claim const worker = ownEntry(state.projectHarnessWorkers ?? {}, claim?.workerId) const sdk = sameTask(worker, task) || worker && attempt && worker.projectId === task.projectId && worker.attemptId === attempt.id ? worker : null + const nativeCandidate = ownEntry(state.projectNativeWorkers ?? {}, claim?.workerId) + const nativeWorker = sameTask(nativeCandidate, task) ? nativeCandidate : null + const executionView = executionMetadataView(state, sdk ?? nativeWorker) + const nativeActivated = nativeWorker && isIntentCommitted(nativeWorker.activationEffect) const native = ownEntry(state.bots ?? {}, claim?.workerId) const nativeActive = native?.projectId === task.projectId && (!attempt || native.attemptId === attempt.id) && activity[claim?.workerId] const execution = attempt?.runId ? ownEntry(state.projectExecutions ?? {}, attempt.runId) : null const run = sameTask(execution, task) ? execution : null - const currentRole = (state.projectAgentDefinitions?.[task.projectId]?.roles ?? []).find(row => row.id === c?.botId)?.role ?? (c?.botId?.startsWith(`${task.projectId}:`) && roles.has(c.botId.slice(task.projectId.length + 1)) ? c.botId.slice(task.projectId.length + 1) : null) + const catalog = roleCatalog ?? state.projectAgentDefinitions?.[task.projectId] + const currentRole = (catalog?.roles ?? []).find(row => row.id === c?.botId)?.role ?? claim?.role ?? Object.values(state.projectRetiredRoles ?? {}).find(row => row.projectId === task.projectId && row.id === c?.botId)?.role ?? (c?.botId?.startsWith(`${task.projectId}:`) && roles.has(c.botId.slice(task.projectId.length + 1)) ? c.botId.slice(task.projectId.length + 1) : null) let phase = 'published', waitingFor = null, reason = null if (task.status === 'accepted') phase = 'accepted' else if (task.status === 'cancelled' || c?.status === 'cancelled' || attempt?.status === 'cancelled') phase = 'cancelled' else if (attempt?.cancelRequested || c?.status === 'stopping') { phase = 'stopping'; waitingFor = 'worker-exit' } + else if (nativeWorker?.status === 'stopping' || nativeWorker?.status === 'stop-unconfirmed') { phase = nativeWorker.status === 'stopping' ? 'stopping' : 'blocked'; waitingFor = 'worker-exit'; reason = c?.block?.reason ?? null } else if (task.chiefQuestion) { phase = 'awaiting-owner'; waitingFor = task.chiefQuestion.recipient?.kind === 'personal-agent' ? 'personal-agent' : 'owner'; reason = task.chiefQuestion.reason } else if (task.projectTask?.admission.status !== 'accepted' && task.projectTask) { phase = 'awaiting-chief'; waitingFor = 'admission' } else if (c?.status === 'waiting-children') { phase = 'blocked'; waitingFor = 'children'; reason = c.block?.reason ?? null } @@ -57,9 +66,10 @@ export function taskView(state, task, { activity = {} } = {}) { else if (c?.status === 'blocked' || queue?.status === 'blocked') { phase = 'blocked'; reason = c?.block?.reason ?? queue?.reason ?? null } else if (queue) { phase = queue.status === 'starting' ? 'starting' : 'queued'; waitingFor = queue.reason ? 'resource' : 'capacity'; reason = queue.reason ?? null } else if (c?.status === 'queued') { phase = 'queued'; waitingFor = 'role' } + else if (c?.status === 'claimed' && executionView?.exitConfirmed) { phase = 'stopping'; waitingFor = 'handoff' } else if (c?.status === 'claimed') { if (sdk?.status === 'waiting-input') { phase = 'awaiting-input'; waitingFor = 'owner' } - else if (claim?.status === 'active' && (sdk?.status === 'running' || !sdk && nativeActive)) phase = 'running' + else if (claim?.status === 'active' && (sdk?.status === 'running' || !sdk && nativeActive && (!nativeWorker || nativeActivated))) phase = 'running' else { phase = 'starting'; waitingFor = 'execution' } } else if (task.projectTask?.block && task.status === 'blocked') { phase = 'blocked'; waitingFor = 'chief'; reason = task.projectTask.block.reason } @@ -75,11 +85,11 @@ export function taskView(state, task, { activity = {} } = {}) { } const tests = attempt?.evidence?.tests, review = attempt?.evidence?.review const progress = inspectionFacts(state, { ...task, collaboration: c ? { ...c, history: c.history ?? [] } : undefined, ...(attempt ? {} : { deliveryAttemptId: null }) }, { maxHandoffsWithoutProgress: Infinity, longExecutionMs: Infinity, missingHeartbeatMs: Infinity, longWaitMs: Infinity }) - return { version: DEVELOPMENT_PROTOCOL, projectId: task.projectId, taskId: task.id, publication: 'published', phase, waitingFor, reason, outcome: taskOutcome(state, task), - botId: c?.botId ?? null, currentRole, generation: c?.generation ?? null, attemptId: attempt?.id ?? null, - contract: task.projectTask ? structuredClone(task.projectTask) : null, + return { version: DEVELOPMENT_PROTOCOL, projectId: task.projectId, taskId: task.id, title: typeof task.title === 'string' ? task.title.slice(0, 1000) : '', source: taskProvenance(task), publication: 'published', phase, waitingFor, reason, outcome: taskOutcome(state, task), + botId: c?.botId ?? null, currentRole, roleDefinitionRevision: catalog?.revision ?? 1, execution: executionView, generation: c?.generation ?? null, attemptId: attempt?.id ?? null, + contract: taskContractView(task), executionId: claim?.id ?? run?.id ?? null, workerId: claim?.workerId ?? null, queueId: queue?.id ?? null, candidateSha: sha, - executionGeneration: sdk?.generation ?? null, inputRequestId: sdk?.pending?.id ?? null, + executionGeneration: sdk?.generation ?? nativeWorker?.generation ?? null, inputRequestId: sdk?.pending?.id ?? null, lastMeaningfulProgress: progress.lastMeaningfulProgressAt === null ? null : { at: progress.lastMeaningfulProgressAt, evidence: progress.progressEvidence }, evidence: { tests: tests ? { sha: tests.sha, sha256: tests.sha256, passed: tests.passed === true, current: Boolean(sha && tests.sha === sha) } : null, review: review ? { sha: review.sha, verdict: review.verdict, current: Boolean(sha && review.sha === sha) } : null }, @@ -88,8 +98,13 @@ export function taskView(state, task, { activity = {} } = {}) { } export function createDevelopmentService(rt) { + const receipts = rt.projectEffects ? createCommandReceipts(rt) : null + const management = new Set(['role.decide', 'attempt.stop', 'attempt.cleanup', 'queue.retry', 'chief.inspect', 'chief.configure', 'schedule.configure', 'effect.retry', 'attempt.integration-bind', 'attempt.publication-retry', 'source.subscribe', 'source.disable', 'source.poll']) + const readCommand = new Set(['agent.mode', 'consultation.ask', 'consultation.reply']) + const actionFor = name => ['task.propose', 'proposal.publish', 'comment.publish'].includes(name) ? 'create' : management.has(name) ? 'configure' : readCommand.has(name) ? 'read' : name === 'task.accept' || name === 'task.reject' ? 'accept' : 'task-write' + function access(actor) { - if (!rt.sharedProjectMode?.() || !actor?.memberId || !ownEntry(rt.state.members ?? {}, actor.memberId)) throw failure('Sign in to the Project service', 'bot/unauthorized') + if (!rt.sharedProjectMode?.() || !actor?.memberId || !ownEntry(rt.state.members ?? {}, actor.memberId) || ownEntry(rt.state.members ?? {}, actor.memberId)?.status === 'disabled') throw failure('Sign in to the Project service', 'bot/unauthorized') } function checkedTask(id, projectId, actor) { const task = ownEntry(rt.state.tasks, String(id ?? '')) @@ -99,7 +114,7 @@ export function createDevelopmentService(rt) { } function view(task, actor) { rt.requireProjectAccess(actor, task.projectId) - const result = taskView(rt.state, task, { activity: rt.activityView?.() ?? {} }) + const result = taskView(rt.state, task, { activity: rt.activityView?.() ?? {}, roleCatalog: rt.projectRoleGovernance?.visibleSet(task.projectId) ?? ownEntry(rt.state.projectAgentDefinitions ?? {}, task.projectId), isIntentCommitted: token => Boolean(rt.projectEffects?.intentCommitted(token)) }) const allowed = action => rt.projectCan(actor, task.projectId, action, task) if (result.phase === 'published' || result.phase === 'failed') { if (allowed('create') && allowed('task-write')) result.nextActions.push('task.run') } if (result.queueId && ['queued', 'blocked'].includes(result.phase) && allowed('task-write')) result.nextActions.push('queue.cancel') @@ -113,16 +128,20 @@ export function createDevelopmentService(rt) { return result } function attemptView(attempt) { - const policy = row => row ? { version: row.version, role: row.role, commit: row.commit, profile: row.profile, sha256: row.sha256 } : null - const session = row => { - const { context, policy: rules, ...rest } = row - return { ...rest, policy: policy(rules), context: context ? { version: context.version, sha256: context.sha256, archiveSha256: context.archiveSha256, ...(context.budget ? { budget: context.budget, usage: context.usage } : {}) } : null } + const policy = row => row ? { version: row.version, role: row.role, policyRole: row.policyRole ?? row.role, commit: row.commit, profile: row.profile, sha256: row.sha256 } : null + const session = (row, section = 'roleSessions') => { + const { context, policy: rules } = row + const keys = ['id', 'projectId', 'taskId', 'attemptId', 'role', 'policyRole', 'logicalSessionId', 'definitionRef', 'definitionRevision', 'definitionKey', 'generation', 'poolClaimId', 'status', 'botId', 'sessionId', 'backendSessionId'] + return { ...Object.fromEntries(keys.filter(key => row[key] !== undefined).map(key => [key, structuredClone(row[key])])), + execution: row.execution ? { harness: row.execution.harness, model: row.execution.model ?? null, resourceIds: (row.execution.resourceIds ?? []).slice(0, 20) } : null, + definitionSource: { query: 'attempt.context', attemptId: attempt.id, section }, + policy: policy(rules), context: context ? { version: context.version, sha256: context.sha256, archiveSha256: context.archiveSha256, ...(context.budget ? { budget: context.budget, usage: context.usage } : {}) } : null } } - const { roleSessions, roleSessionHistory, rolePolicies, ...rest } = attempt - return structuredClone({ ...rest, - ...(roleSessions ? { roleSessions: Object.fromEntries(Object.entries(roleSessions).map(([key, row]) => [key, session(row)])) } : {}), - ...(roleSessionHistory ? { roleSessionHistory: roleSessionHistory.map(session) } : {}), - ...(rolePolicies ? { rolePolicies: Object.fromEntries(Object.entries(rolePolicies).map(([key, row]) => [key, policy(row)])) } : {}), + const { roleSessions, roleSessionHistory, rolePolicies } = attempt + return structuredClone({ ...attemptMetadataView(attempt), + ...(roleSessions ? { roleSessions: Object.fromEntries(Object.entries(roleSessions).slice(0, 32).map(([key, row]) => [key, session(row)])) } : {}), + ...(roleSessionHistory ? { roleSessionHistory: roleSessionHistory.slice(-3).map(row => session(row, 'roleSessionHistory')), roleSessionHistoryPage: { total: roleSessionHistory.length, shown: Math.min(3, roleSessionHistory.length), source: { query: 'attempt.context', attemptId: attempt.id, section: 'roleSessionHistory' } } } : {}), + ...(rolePolicies ? { rolePolicies: Object.fromEntries(Object.entries(rolePolicies).slice(0, 32).map(([key, row]) => [key, policy(row)])) } : {}), contextRef: { query: 'attempt.context', attemptId: attempt.id, sections: ['roleSessions', 'roleSessionHistory', 'rolePolicies'] } }) } function detail(body, actor) { @@ -132,11 +151,11 @@ export function createDevelopmentService(rt) { const end = body.beforeAttemptId === undefined ? rows.length : rows.findIndex(row => row.id === body.beforeAttemptId) if (end < 0) throw failure('Attempt cursor is unavailable') const page = rows.slice(Math.max(0, end - limit), end) - return { task: structuredClone(task), attempts: page.map(attemptView), view: view(task, actor), + return { task: taskDetailView(task), attempts: page.map(attemptView), view: view(task, actor), attemptsPage: { total: rows.length, limit, before: end > limit ? page[0].id : null } } } function context(body) { - if (!['roleSessions', 'roleSessionHistory', 'rolePolicies'].includes(body.section)) throw failure('Unknown Attempt context section') + if (!['roleSessions', 'roleSessionHistory', 'rolePolicies', 'evidence', 'feedback', 'requirement', 'failure', 'agentDefinitions', 'integrationAdmissions'].includes(body.section)) throw failure('Unknown Attempt context section') const attempt = ownEntry(rt.state.projectAttempts ?? {}, body.attemptId) const text = JSON.stringify(attempt[body.section] ?? null), offset = body.offset ?? 0, limit = body.limit ?? 8000 if (!Number.isSafeInteger(offset) || offset < 0 || offset > text.length || !Number.isSafeInteger(limit) || limit < 1 || limit > 16000) throw failure('Invalid context page') @@ -153,6 +172,11 @@ export function createDevelopmentService(rt) { return { run: await rt.projectChiefControl.request(body.projectId, 'inspection', { requestId: body.requestId, requestedBy: actor.memberId, ...(task ? { blockedTaskId: task.id } : {}) }) } } async function invoke(endpoint, body, actor) { + if (endpoint === 'task-create') { + let botId = body.botId + if (!botId) botId = (await rt.projectAgents.ensurePublic(body.projectId, actor)).botId + return rt.taskEndpoint(endpoint, { ...body, botId }, actor) + } if (endpoint === 'project-execution-boundary') return rt.projectExecutionBoundary(body, actor) if (endpoint === 'project-sources') return rt.projectSources.query(body, actor) if (endpoint === 'project-source-subscribe') return rt.projectSources.subscribe(body, actor) @@ -172,6 +196,7 @@ export function createDevelopmentService(rt) { if (endpoint === 'project-scheduled-task') return rt.projectScheduledTasks.endpoint(body, actor) if (endpoint === 'project-agent-mode') return rt.projectDelegations.mode(body, actor) if (endpoint === 'project-execution-input') return rt.projectHarness.input(body, actor) + if (['project-roles', 'project-role-proposals', 'project-role-decision'].includes(endpoint)) return rt.projectAgents.endpoint(endpoint, body, actor) if (endpoint === 'project-agent-proposal-apply') return rt.projectAgents.endpoint(endpoint, body, actor) if (endpoint === 'project-task-detail') return detail(body, actor) if (['project-consultation-reply', 'project-consultation-ask', 'project-consultation-get', 'project-consultation-list'].includes(endpoint)) return rt.projectAgents.consultation(endpoint, body, actor) @@ -183,18 +208,25 @@ export function createDevelopmentService(rt) { } return rt.projectDelivery.endpoint(endpoint, body, actor) } + async function commandResult(name, args, actor) { + const result = await invoke(operations[name], args, actor) + if (name === 'task.propose') return taskDetailView(result) + if (!record(result)) return result + return { ...result, ...(result.attempt ? { attempt: attemptView(result.attempt) } : {}), ...(result.task ? { task: taskDetailView(result.task) } : {}), + ...(result.run ? { run: Object.fromEntries(['id', 'projectId', 'taskId', 'kind', 'status', 'startedAt', 'completedAt', 'createdAt'].filter(key => result.run[key] !== undefined).map(key => [key, result.run[key]])), runRef: { query: 'chief.state', projectId: args.projectId } } : {}) } + } async function envelope(body, actor) { if (!record(body) || body.version !== DEVELOPMENT_PROTOCOL || !['command', 'query'].includes(body.kind) || !record(body.payload ?? {})) throw failure('Unsupported Project development request') const { name, projectId } = body rt.requireProjectAccess(actor, projectId) - if (!Object.hasOwn(operations, name) && !['tasks', 'task', 'attempt.context', 'effects'].includes(name)) throw failure('Unknown Project development operation') + if (!Object.hasOwn(operations, name) && !['tasks', 'task', 'task.history', 'project.snapshot', 'events', 'command.receipt', 'command.result', 'attempt.context', 'effects'].includes(name)) throw failure('Unknown Project development operation') if (queries.has(name) !== (body.kind === 'query')) throw failure('Operation kind does not match the contract') if (body.kind === 'command' && !stableId(body.requestId)) throw failure('A command needs a stable requestId') const args = { ...(body.payload ?? {}), projectId, ...(body.requestId !== undefined ? { requestId: body.requestId } : {}) } if (body.payload?.projectId !== undefined && body.payload.projectId !== projectId) throw failure('Payload belongs to another Project', 'bot/forbidden') // These targets have their own authority in the existing services, but the // envelope must also reject a target in another Project the actor can access. - for (const [key, table] of [['proposalId', rt.state.projectAgentProposals], ['workerId', rt.state.projectHarnessWorkers]]) { + for (const [key, table] of [['proposalId', ['role.decide', 'role.proposals'].includes(name) ? rt.state.projectRoleProposals : rt.state.projectAgentProposals], ['workerId', rt.state.projectHarnessWorkers]]) { if (args[key] === undefined) continue const target = ownEntry(table ?? {}, String(args[key])) if (!target || target.projectId !== projectId) throw failure('Target is unavailable in this Project', 'bot/forbidden') @@ -210,10 +242,26 @@ export function createDevelopmentService(rt) { if (task && selected.id !== task.id || args.taskId && selected.id !== args.taskId) throw failure('Targets belong to different Tasks', 'bot/forbidden') task = selected } - if (body.expectedGeneration !== undefined && (!Number.isSafeInteger(body.expectedGeneration) || body.expectedGeneration < 0 || !task || task.collaboration?.generation !== body.expectedGeneration)) throw failure('Task generation changed', 'bot/conflict') - if (body.expectedCandidateSha !== undefined && (body.expectedCandidateSha !== null && (typeof body.expectedCandidateSha !== 'string' || !/^[a-f0-9]{40}$/.test(body.expectedCandidateSha)) || !task || taskView(rt.state, task).candidateSha !== body.expectedCandidateSha)) throw failure('Task candidate changed', 'bot/conflict') + if (!task && name === 'execution.input') { + const worker = ownEntry(rt.state.projectHarnessWorkers ?? {}, args.workerId) + if (worker) task = checkedTask(worker.taskId ?? ownEntry(rt.state.projectAttempts ?? {}, worker.attemptId)?.taskId, projectId, actor) + } + const check = () => { + if (task) task = checkedTask(task.id, projectId, actor) + if (body.expectedGeneration !== undefined && (!Number.isSafeInteger(body.expectedGeneration) || body.expectedGeneration < 0 || !task || task.collaboration?.generation !== body.expectedGeneration)) throw failure('Task generation changed', 'bot/conflict') + if (body.expectedCandidateSha !== undefined && (body.expectedCandidateSha !== null && (typeof body.expectedCandidateSha !== 'string' || !/^[a-f0-9]{40}$/.test(body.expectedCandidateSha)) || !task || taskView(rt.state, task).candidateSha !== body.expectedCandidateSha)) throw failure('Task candidate changed', 'bot/conflict') + } + if (body.kind === 'query') check() let result - if (name === 'effects') result = { effects: rt.projectEffects.list(projectId, actor, args) } + if (name === 'command.receipt') { if (!receipts) throw failure('Command receipts are unavailable'); result = receipts.get(projectId, args.commandRequestId, actor, args.commandName) } + else if (name === 'command.result') { if (!receipts) throw failure('Command receipts are unavailable'); result = receipts.result(projectId, args, actor) } + else if (name === 'events') result = rt.committedProjectEvents.query(projectId, args) + else if (name === 'project.snapshot') result = rt.committedProjectEvents.snapshot(projectId, args) + else if (name === 'effects') result = { effects: rt.projectEffects.list(projectId, actor, args) } + else if (name === 'task.history') { + const source = checkedTask(args.taskId, projectId, actor) + result = { taskId: source.id, section: args.section, ...sourcePage(taskHistorySource(source, args.section), args, text => createHash('sha256').update(text).digest('hex')) } + } else if (name === 'task') result = { task: view(checkedTask(args.taskId, projectId, actor), actor) } else if (name === 'attempt.context') { if (!args.attemptId) throw failure('Attempt context needs an Attempt') @@ -227,10 +275,20 @@ export function createDevelopmentService(rt) { if (args.after !== undefined && index < 0) throw failure('Task cursor is unavailable') const page = rows.slice(index + 1, index + 1 + limit) result = { tasks: page.map(row => view(row, actor)), next: index + 1 + page.length < rows.length ? page.at(-1).id : null } - } else result = await invoke(operations[name], args, actor) + } else if (body.kind === 'command' && receipts) { + const delivered = await receipts.run({ projectId, name, requestId: body.requestId, args, actor, action: actionFor(name), taskId: task?.id, + expectedGeneration: body.expectedGeneration, expectedCandidateSha: body.expectedCandidateSha, + reconcile: name === 'task.run' ? async () => ({ result: await commandResult(name, args, actor) }) : ['task.propose', 'proposal.publish'].includes(name) ? async row => { + const existing = Object.values(rt.state.tasks).find(task => task.projectId === projectId && task.commandSource?.commandReceiptId === row.id) + const proof = existing && Object.values(rt.state.projectEffects ?? {}).find(effect => effect.kind === 'task.intake' && effect.taskId === existing.id && rt.projectEffects.intentCommitted({ id: effect.id, generation: effect.generation })) + return proof ? { result: taskDetailView(existing) } : null + } : undefined, + }, () => commandResult(name, args, actor), check) + return { version: DEVELOPMENT_PROTOCOL, requestId: body.requestId, ...delivered } + } else { check(); result = await invoke(operations[name], args, actor) } return { version: DEVELOPMENT_PROTOCOL, ...(body.requestId !== undefined ? { requestId: body.requestId } : {}), result } } - return { view: (task, actor) => { access(actor); return view(task, actor) }, + return { commandSource: projectId => receipts?.source(projectId) ?? null, view: (task, actor) => { access(actor); return view(task, actor) }, isQuery: developmentQuery, // Trusted host ports. Public HTTP never accepts role tokens or model results. roleResult: row => rt.projectPools.result(row), diff --git a/patches/ds-bot/files/src/host/project-effects.js b/patches/ds-bot/files/src/host/project-effects.js index 5a13aece..96b0b996 100644 --- a/patches/ds-bot/files/src/host/project-effects.js +++ b/patches/ds-bot/files/src/host/project-effects.js @@ -1,5 +1,6 @@ // Persist intent with Project state, then execute through registered host adapters. import { createHash } from 'node:crypto' +import { commandEffectVisible } from './project-command-receipts.js' import { ownEntry, setEntry } from './store.js' export const EFFECT_PROTOCOL = 'project-effect/v1' @@ -177,13 +178,14 @@ export function createProjectEffects(rt, { now = Date.now } = {}) { function list(projectId, actor, { taskId, limit = 50 } = {}) { rt.requireProjectAccess(actor, projectId) if (!Number.isSafeInteger(limit) || limit < 1 || limit > 200) throw fail('Effect page limit must be between 1 and 200') - return Object.values(rt.state.projectEffects ?? {}).filter(row => row.projectId === projectId && (!taskId || row.taskId === taskId)).sort((a, b) => b.updatedAt - a.updatedAt || a.id.localeCompare(b.id)).slice(0, limit) + return Object.values(rt.state.projectEffects ?? {}).filter(row => row.projectId === projectId && commandEffectVisible(rt.state, row, actor) && (!taskId || row.taskId === taskId)).sort((a, b) => b.updatedAt - a.updatedAt || a.id.localeCompare(b.id)).slice(0, limit) .map(({ payload: _payload, history: _history, key: _key, ...row }) => ({ ...copy(row), committed: durable.get(row.id) === stamp(row) })) } async function retry(body, actor) { const row = ownEntry(rt.state.projectEffects ?? {}, body.effectId) if (!row || row.projectId !== body.projectId) throw fail('Effect is unavailable in this Project') rt.requireProjectAccess(actor, row.projectId, 'configure') + if (!commandEffectVisible(rt.state, row, actor)) throw fail('Effect is private to another command principal') if (rt.readOnly?.() !== undefined || !/^[A-Za-z0-9_-]{1,100}$/.test(body.requestId ?? '')) throw fail('Effect retry needs a writable host and stable requestId') if (row.version !== EFFECT_PROTOCOL) throw fail('Effect protocol upgrade is required') const prior = row.retries?.find(item => item.requestId === body.requestId) diff --git a/patches/ds-bot/files/src/host/project-pools.js b/patches/ds-bot/files/src/host/project-pools.js index 7db6d0d9..26dc2117 100644 --- a/patches/ds-bot/files/src/host/project-pools.js +++ b/patches/ds-bot/files/src/host/project-pools.js @@ -55,9 +55,16 @@ export function handoffContract(state, task, receipt) { } export function createProjectPools(rt) { - let adapter, tail = Promise.resolve() + let adapter, acceptanceEffects, tail = Promise.resolve() let recovered = false const serial = fn => { const result = tail.then(fn, fn); tail = result.catch(() => {}); return result } + function acceptanceAdapter() { + if (rt.projectEffects && acceptanceEffects !== rt.projectEffects) { + rt.projectEffects.register('pool.accept', { run: effect => ({ claimId: effect.payload.claimId, handoffId: effect.payload.handoffId, status: 'durable-acceptance' }) }, { replace: true }) + acceptanceEffects = rt.projectEffects + } + } + acceptanceAdapter() // Register on restart even when no new claim has been made. const taskOf = id => { const task = ownEntry(rt.state.tasks, id); if (!task?.projectId) throw refuse('Unknown Project Task'); return task } const definitions = projectId => rt.projectAgents.deliveryDefinitions(projectId) const targetOf = (projectId, botId) => { @@ -86,6 +93,7 @@ export function createProjectPools(rt) { async function current(task, token) { const claim = task.collaboration?.claim if (!claim || claim.id !== token?.id || claim.generation !== token?.generation || claim.botId !== token?.botId || claim.status !== 'active') throw refuse('Claim is stale or belongs to another Bot') + if (claim.acceptanceEffect && !rt.projectEffects?.intentCommitted(claim.acceptanceEffect)) throw refuse('Claim acceptance has no confirmed durable intent') const requirementsRevision = claim.requirementsRevision ?? rt.state.projectAttempts?.[claim.attemptId]?.requirement?.projectTaskRevision const currentRequirements = () => !task.projectTask || requirementsRevision === undefined || requirementsRevision === task.projectTask.requirements.revision if (!currentRequirements()) throw refuse('Task requirements version changed before execution decision') @@ -198,6 +206,7 @@ export function createProjectPools(rt) { await authority(task) const prior = c?.history.find(row => row.acceptance?.requestId === requestId) if (prior) { + if (prior.acceptance.effect && !rt.projectEffects?.intentCommitted(prior.acceptance.effect)) throw refuse('Handoff acceptance is not durably committed') if (prior.id !== handoffId || prior.to !== botId) throw refuse('Accept idempotency key was reused for another handoff') const execution = prior.execution ?? (c.claim?.id === prior.claimId ? c.claim : null) return execution ? clone(execution) : { ...clone(prior.acceptance), id: prior.claimId, status: 'historical' } @@ -231,11 +240,33 @@ export function createProjectPools(rt) { const limit = rt.config.projectHandoff?.quota?.steps if (Number.isFinite(limit) && (c.usage?.steps ?? 0) >= limit) { c.status = 'blocked'; c.block = { reason: 'Handoff refused (quota): execution step quota is exhausted', at: Date.now() }; await rt.save(); throw refuse(c.block.reason) } const claim = { id: `claim-${randomUUID()}`, botId, role: definition.role, policyRole: definition.policyRole ?? definition.role, ...(task.projectTask ? { requirementsRevision: task.projectTask.requirements.revision } : {}), generation: ++c.generation, taskId: task.id, projectId: task.projectId, attemptId: c.attemptId, status: 'active', createdAt: Date.now() } + const before = { generation: expectedGeneration, status: c.status, claim: c.claim, usage: c.usage === undefined ? undefined : clone(c.usage), receipt: clone(receipt), wait: task.projectTask?.wait ? clone(task.projectTask.wait) : null } c.usage ??= {}; c.usage.steps = (c.usage.steps ?? 0) + 1 if (receipt.purpose === 'children-completed' && task.projectTask?.wait?.status === 'queued') Object.assign(task.projectTask.wait, { status: 'resumed', resumedClaimId: claim.id, resumedAt: Date.now() }) c.claim = claim; c.status = 'claimed'; receipt.status = 'accepted'; receipt.acceptedAt = Date.now(); receipt.claimId = claim.id receipt.acceptance = { claimId: claim.id, botId: claim.botId, generation: claim.generation, acceptedAt: receipt.acceptedAt, ...(requestId ? { requestId } : {}), ...(claim.chiefRunId ? { chiefRunId: claim.chiefRunId } : {}) } - await rt.save() + let intent + try { + if (rt.projectEffects) { + acceptanceAdapter() + intent = rt.projectEffects.stage({ kind: 'pool.accept', key: claim.id, projectId: task.projectId, taskId: task.id, + payload: { claimId: claim.id, handoffId: receipt.id, attemptId: claim.attemptId, botId: claim.botId, generation: claim.generation } }) + claim.acceptanceEffect = intent + receipt.acceptance.effect = intent + await rt.projectEffects.persist(intent) + } else await rt.save() // In-memory/legacy hosts without the durable Effects service. + } catch (error) { + const uncommitted = !intent || rt.projectEffects.discardUncommitted(intent) + if (uncommitted && c.claim === claim && c.generation === claim.generation && c.currentHandoffId === receipt.id) { + c.generation = before.generation; c.status = before.status + if (before.claim === undefined) delete c.claim; else c.claim = before.claim + if (before.usage === undefined) delete c.usage; else c.usage = before.usage + for (const key of Object.keys(receipt)) delete receipt[key] + Object.assign(receipt, before.receipt) + if (before.wait) task.projectTask.wait = before.wait + } + throw error + } return clone(claim) } async function decide(taskId, token, decision) { diff --git a/patches/ds-bot/files/src/host/project-role-governance.js b/patches/ds-bot/files/src/host/project-role-governance.js index 1757d0d2..a1077a29 100644 --- a/patches/ds-bot/files/src/host/project-role-governance.js +++ b/patches/ds-bot/files/src/host/project-role-governance.js @@ -16,7 +16,7 @@ export function createProjectRoleGovernance(rt, ports) { const previousSets = new Map() rt.projectEffects?.register('role.governance', { run: effect => ({ proposalId: effect.payload.proposalId ?? null, changeId: effect.payload.changeId ?? null, status: 'durable-role-record' }) }) const committed = token => Boolean(token && rt.projectEffects?.intentCommitted(token)) - function visibleSet(projectId, current = ports.definitionSet(projectId)) { + function visibleSet(projectId, current = ports.readDefinitionSet ? ports.readDefinitionSet(projectId) : ports.definitionSet(projectId)) { if (!current.changeEffect || committed(current.changeEffect)) return current const previous = previousSets.get(projectId) if (!previous) throw fail('Role definition change is not committed', 'bot/conflict') diff --git a/patches/ds-bot/files/src/host/project-service-events.js b/patches/ds-bot/files/src/host/project-service-events.js new file mode 100644 index 00000000..2e455d55 --- /dev/null +++ b/patches/ds-bot/files/src/host/project-service-events.js @@ -0,0 +1,136 @@ +import { executionMetadataView } from './project-service-views.js' +// A committed read model over the original Store. No event reception grants +// initiative, and private conversations/credentials never enter this projection. +import { createHash } from 'node:crypto' +const hash = value => createHash('sha256').update(JSON.stringify(value)).digest('hex') +const copy = value => structuredClone(value) +const own = (table, id) => table && Object.hasOwn(table, id) ? table[id] : undefined +const text = value => typeof value === 'string' ? value.slice(0, 1000) : '' +const fail = (message, code = 'bot/invalid') => Object.assign(new Error(message), { code }) +const VERSION = 'project-event/v1' +function projections(state) { + const rows = new Map() + for (const row of Object.values(state.projects ?? {})) { + if (!row?.id) continue + const data = { id: row.id, name: text(row.name), status: row.status, goal: text(row.goal), scope: text(row.scope), repository: row.repository ?? null, roleDefinitionRevision: own(state.projectAgentDefinitions, row.id)?.revision ?? 1 } + rows.set(`project:${row.id}`, { projectId: row.id, entityType: 'project', id: row.id, data, fingerprint: hash([data, row.roles ?? {}, row.repositories ?? []]) }) + } + for (const row of Object.values(state.tasks ?? {})) { + if (!row?.projectId || !own(state.projects, row.projectId)) continue + const c = row.collaboration, data = { id: row.id, projectId: row.projectId, title: text(row.title), status: row.status, + generation: c?.generation ?? null, collaborationStatus: c?.status ?? null, claimId: c?.claim?.id ?? null, workerId: c?.claim?.workerId ?? null, botId: c?.botId ?? null, attemptId: row.deliveryAttemptId ?? null, + admission: row.projectTask?.admission.status ?? null, requirementsRevision: row.projectTask?.requirements.revision ?? null, + waitStatus: row.projectTask?.wait?.status ?? null, block: Boolean(c?.block || row.projectTask?.block), resultDecisionId: row.completedBy?.decisionId ?? null } + rows.set(`task:${row.id}`, { projectId: row.projectId, entityType: 'task', id: row.id, data, fingerprint: hash([data, row.note ?? '', row.criteria ?? '', row.projectTask?.requirements ?? null]) }) + } + for (const row of Object.values(state.projectAttempts ?? {})) { + const task = own(state.tasks, row.taskId) + if (!task || task.projectId !== row.projectId || !own(state.projects, row.projectId)) continue + const data = { id: row.id, taskId: row.taskId, projectId: row.projectId, status: row.status, candidateSha: row.candidateSha ?? null, + tests: row.evidence?.tests ? { sha: row.evidence.tests.sha, passed: row.evidence.tests.passed === true } : null, + review: row.evidence?.review ? { sha: row.evidence.review.sha, verdict: row.evidence.review.verdict } : null, + publication: row.pr ? { number: row.pr.number, sha: row.pr.sha } : null, stopping: Boolean(row.cancelRequested) } + rows.set(`attempt:${row.id}`, { projectId: row.projectId, entityType: 'attempt', id: row.id, data, fingerprint: hash(data) }) + } + for (const row of Object.values(state.projectRoleProposals ?? {})) { + if (!row?.projectId || !own(state.projects, row.projectId)) continue + const data = { id: row.id, projectId: row.projectId, role: row.role, action: row.action, status: row.status, baseRevision: row.baseRevision, createdAt: row.createdAt ?? null, decisionAt: row.decision?.at ?? null } + rows.set(`role-proposal:${row.id}`, { projectId: row.projectId, entityType: 'role-proposal', id: row.id, data, fingerprint: hash(data) }) + } + for (const [namespace, entityType] of [['projectHarnessWorkers', 'execution'], ['projectNativeWorkers', 'execution'], ['projectResourceLeases', 'resource'], ['projectTaskInputs', 'input']]) { + for (const row of Object.values(state[namespace] ?? {})) { + const attempt = own(state.projectAttempts, row.attemptId) + const task = own(state.tasks, row.taskId ?? attempt?.taskId) + if (!task || task.projectId !== row.projectId || row.chiefRunId || !own(state.projects, row.projectId)) continue + const data = { id: row.id, projectId: row.projectId, taskId: task.id, attemptId: row.attemptId ?? null, status: row.status, + ...(entityType === 'execution' ? executionMetadataView(state, row) : entityType === 'input' ? { kind: row.kind, sha256: row.sha256, deliveries: row.deliveries?.length ?? 0, readConfirmed: row.deliveries?.some(delivery => Boolean(delivery.readAt)) ?? false } : { role: row.role ?? null, resourceId: row.resourceId ?? null, generation: row.generation ?? null, pending: row.pending ? { id: row.pending.id, kind: row.pending.kind } : null }) } + rows.set(`${entityType}:${row.id}`, { projectId: row.projectId, entityType, id: row.id, data, fingerprint: hash(data) }) + } + } + return rows +} +export function createProjectServiceEvents({ maxPerProject = 1024, maxAgeMs = 24 * 60 * 60 * 1000, now = Date.now } = {}) { + if (!Number.isSafeInteger(maxPerProject) || maxPerProject < 1 || maxPerProject > 4096 || !Number.isSafeInteger(maxAgeMs) || maxAgeMs < 1000) throw fail('Invalid Project event retention policy') + let committed = { projections: new Map(), streams: {}, events: {} } + const epoch = (state, projectId) => hash([VERSION, state.workspaceId ?? 'legacy', projectId, own(state.projects, projectId)?.createdAt ?? 0]).slice(0, 32) + function streamFor(state, projectId, streams) { + const prior = own(streams, projectId) + if (prior) return prior + const stream = { epoch: epoch(state, projectId), nextSequence: 1, floor: 0, revision: 0, lastAt: 0 } + Object.defineProperty(streams, projectId, { value: stream, enumerable: true, writable: true, configurable: true }) + return stream + } + function load(state) { + const streams = copy(state.projectServiceEventStreams ?? {}), events = copy(state.projectServiceEvents ?? {}) + for (const row of Object.values(state.projects ?? {})) if (row?.id) streamFor(state, row.id, streams) + for (const [projectId, stream] of Object.entries(streams)) if (!/^[a-f0-9]{32}$/.test(stream.epoch) || !Number.isSafeInteger(stream.nextSequence) || !Number.isSafeInteger(stream.floor) || stream.floor < 0 || stream.nextSequence <= stream.floor || !Number.isSafeInteger(stream.revision) || stream.revision < 0 || stream.lastAt !== undefined && (!Number.isSafeInteger(stream.lastAt) || stream.lastAt < 0)) throw fail(`Project event stream metadata is malformed: ${projectId}`) + for (const row of Object.values(events)) if (row.version !== VERSION || !own(streams, row.projectId) || row.epoch !== streams[row.projectId].epoch || !Number.isSafeInteger(row.sequence) || row.sequence <= streams[row.projectId].floor || row.sequence >= streams[row.projectId].nextSequence) throw fail('Project event record is outside its committed stream') + for (const [projectId, stream] of Object.entries(streams)) { + const rows = Object.values(events).filter(row => row.projectId === projectId).sort((a, b) => a.sequence - b.sequence) + if (rows.length !== stream.nextSequence - stream.floor - 1 || rows.some((row, index) => row.sequence !== stream.floor + index + 1 || !Number.isSafeInteger(row.at) || row.at < 0 || index && row.at < rows[index - 1].at)) throw fail('Committed Project events have a gap, duplicate or invalid timestamp') + stream.lastAt = Math.max(stream.lastAt ?? 0, rows.at(-1)?.at ?? 0) + } + committed = { projections: projections(state), streams, events } + } + function prepare(snapshot) { + const nextProjections = projections(snapshot), streams = copy(committed.streams), events = copy(committed.events), changes = [], touched = new Set() + for (const [key, row] of nextProjections) if (committed.projections.get(key)?.fingerprint !== row.fingerprint) changes.push({ ...row, operation: 'upsert' }) + for (const [key, row] of committed.projections) if (!nextProjections.has(key)) changes.push({ ...row, operation: 'delete' }) + for (const row of changes) { + const stream = streamFor(snapshot, row.projectId, streams) + if (!touched.has(row.projectId)) { ++stream.revision; touched.add(row.projectId) } + const sequence = stream.nextSequence++, event = { version: VERSION, projectId: row.projectId, epoch: stream.epoch, sequence, + revision: stream.revision, at: Math.max(stream.lastAt ?? 0, now()), type: `${row.entityType}.${row.operation === 'delete' ? 'deleted' : 'changed'}`, entityType: row.entityType, entityId: row.id, + data: row.operation === 'delete' ? { id: row.id } : row.data } + stream.lastAt = event.at + Object.defineProperty(events, hash([row.projectId, stream.epoch, sequence]), { value: event, enumerable: true, writable: true, configurable: true }) + } + for (const [projectId, stream] of Object.entries(streams)) { + const rows = Object.entries(events).filter(([, row]) => row.projectId === projectId).sort((a, b) => a[1].sequence - b[1].sequence) + const retained = rows.filter(([, row]) => row.at >= now() - maxAgeMs).slice(-maxPerProject) + const keep = new Set(retained.map(([key]) => key)) + for (const [key, row] of rows) if (!keep.has(key)) { stream.floor = Math.max(stream.floor, row.sequence); delete events[key] } + } + snapshot.projectServiceEventStreams = streams + snapshot.projectServiceEvents = events + return { snapshot, committed: { projections: nextProjections, streams, events } } + } + function saved(prepared) { committed = prepared.committed } + function cursor(projectId, stream, sequence) { return Buffer.from(JSON.stringify({ v: 1, p: projectId, e: stream.epoch, s: sequence })).toString('base64url') } + function streamOf(projectId) { + const stream = own(committed.streams, projectId) + if (!stream) throw fail('Project has no committed event snapshot', 'bot/not-found') + return stream + } + function query(projectId, { after, limit = 100 } = {}) { + if (!Number.isSafeInteger(limit) || limit < 1 || limit > 100) throw fail('Invalid event page limit') + const stream = streamOf(projectId) + const expired = Object.values(committed.events).filter(row => row.projectId === projectId && row.at < now() - maxAgeMs) + const floor = Math.max(stream.floor, ...expired.map(row => row.sequence)) + let sequence = floor + if (after !== undefined) { + if (typeof after !== 'string' || after.length > 600 || !/^[A-Za-z0-9_-]+$/.test(after)) throw fail('Invalid event cursor') + let parsed + try { parsed = JSON.parse(Buffer.from(after, 'base64url').toString('utf8')) } catch { throw fail('Invalid event cursor') } + if (parsed.v !== 1 || parsed.p !== projectId || !Number.isSafeInteger(parsed.s) || parsed.s < 0) throw fail('Event cursor belongs to another scope', 'bot/conflict') + if (parsed.e !== stream.epoch || parsed.s < floor || parsed.s > stream.nextSequence - 1) throw Object.assign(fail('Event cursor expired; read a current Project snapshot', 'bot/cursor-expired'), { nextAction: 'project.snapshot', details: { nextAction: 'project.snapshot' } }) + sequence = parsed.s + } + const rows = Object.values(committed.events).filter(row => row.projectId === projectId && row.sequence > sequence && row.at >= now() - maxAgeMs).sort((a, b) => a.sequence - b.sequence).slice(0, limit) + return { version: VERSION, projectId, revision: stream.revision, events: rows.map(row => ({ ...copy(row), cursor: cursor(projectId, stream, row.sequence) })), + next: cursor(projectId, stream, rows.at(-1)?.sequence ?? sequence), head: cursor(projectId, stream, stream.nextSequence - 1), oldest: cursor(projectId, stream, floor) } + } + function snapshot(projectId, { after, limit = 50, expectedRevision, entityType = 'task' } = {}) { + const stream = streamOf(projectId) + if (!['task', 'attempt', 'execution', 'resource', 'input', 'role-proposal'].includes(entityType)) throw fail('Unknown public Project snapshot entity type') + if (after !== undefined && expectedRevision === undefined) throw fail('Continuing a Project snapshot requires its expectedRevision', 'bot/conflict') + if (!Number.isSafeInteger(limit) || limit < 1 || limit > 100 || expectedRevision !== undefined && expectedRevision !== stream.revision) throw fail('Project snapshot changed or page limit is invalid', 'bot/conflict') + const rows = [...committed.projections.values()].filter(row => row.projectId === projectId && row.entityType === entityType).sort((a, b) => a.id.localeCompare(b.id)) + const index = after === undefined ? -1 : rows.findIndex(row => row.id === after) + if (after !== undefined && index < 0) throw fail('Project snapshot cursor is unavailable') + const page = rows.slice(index + 1, index + 1 + limit), project = committed.projections.get(`project:${projectId}`) + return { version: VERSION, project: project ? copy(project.data) : null, revision: stream.revision, cursor: cursor(projectId, stream, stream.nextSequence - 1), + ...(entityType === 'task' ? { tasks: page.map(row => copy(row.data)) } : { entityType, entities: page.map(row => copy(row.data)) }), next: index + 1 + page.length < rows.length ? page.at(-1).id : null } + } + return { load, prepare, saved, query, snapshot } +} diff --git a/patches/ds-bot/files/src/host/project-service-sse.js b/patches/ds-bot/files/src/host/project-service-sse.js new file mode 100644 index 00000000..67469aa6 --- /dev/null +++ b/patches/ds-bot/files/src/host/project-service-sse.js @@ -0,0 +1,69 @@ +// Request-scoped transport over authenticated service queries. No background +// project scheduler, bearer-in-URL, raw DSH stream or parallel event authority. +import { setTimeout as delay } from 'node:timers/promises' +import { once } from 'node:events' + +export function createProjectSse({ pull, pollMs = 1000, maxStreams = 64 }) { + if (!Number.isSafeInteger(pollMs) || pollMs < 10 || pollMs > 60000 || !Number.isSafeInteger(maxStreams) || maxStreams < 1 || maxStreams > 256) throw new Error('Invalid Project SSE transport limits') + const active = new Set() + async function handle(req, res, url, respond) { + const token = req.headers['x-flybot-member-session'], projectId = url.searchParams.get('projectId') + const invalid = [...url.searchParams.keys()].some(key => !['projectId', 'after'].includes(key) || url.searchParams.getAll(key).length !== 1) + if (invalid || !/^[A-Za-z0-9_-]{1,100}$/.test(projectId ?? '')) return respond(400, { ok: false, error: { code: 'bot/invalid', message: 'Choose one Project and an optional event cursor' } }) + if (typeof token !== 'string' || !token || token.length > 8192) return respond(401, { ok: false, error: { code: 'bot/unauthorized', message: 'An authenticated member or personal-agent header is required' } }) + let cursor = req.headers['last-event-id'] ?? url.searchParams.get('after') ?? undefined + if (url.searchParams.has('after') && req.headers['last-event-id'] !== undefined && req.headers['last-event-id'] !== url.searchParams.get('after') || cursor !== undefined && (typeof cursor !== 'string' || cursor.length > 600 || !/^[A-Za-z0-9_-]+$/.test(cursor))) return respond(400, { ok: false, error: { code: 'bot/invalid', message: 'Invalid or conflicting event cursor' } }) + if (active.size >= maxStreams) return respond(429, { ok: false, error: { code: 'bot/busy', message: 'Project event transport is at capacity', details: { nextAction: 'retry' } } }) + const controller = new AbortController(), { signal } = controller + active.add(controller) + const close = () => controller.abort() + res.once('close', close) + const frame = (type, data, id) => `${id ? `id: ${id}\n` : ''}event: ${type}\ndata: ${JSON.stringify(data)}\n\n` + function send(type, data, id) { + if (signal.aborted || res.destroyed) return false + return res.write(frame(type, data, id)) + } + try { + const first = await pull({ projectId, name: cursor === undefined ? 'project.snapshot' : 'events', payload: cursor === undefined ? {} : { after: cursor, limit: 100 }, token, signal }) + if (signal.aborted) return + if (cursor === undefined && (first?.project?.id !== projectId || typeof first.cursor !== 'string' || !/^[A-Za-z0-9_-]{1,600}$/.test(first.cursor))) throw new Error('Invalid scoped Project snapshot') + res.writeHead(200, { 'content-type': 'text/event-stream; charset=utf-8', 'cache-control': 'no-store', 'x-accel-buffering': 'no' }) + res.flushHeaders() + let page = first + if (cursor === undefined) { + cursor = first.cursor + if (!send('snapshot', first, cursor)) await once(res, 'drain', { signal }) + page = null + } + while (!signal.aborted) { + page ??= await pull({ projectId, name: 'events', payload: { after: cursor, limit: 100 }, token, signal }) + if (signal.aborted) break + if (!Array.isArray(page.events)) throw new Error('Invalid Project event page') + let backpressured = false + for (const row of page.events) { + if (row.projectId !== projectId || typeof row.cursor !== 'string' || !/^[A-Za-z0-9_-]{1,600}$/.test(row.cursor)) throw new Error('Project event scope changed') + cursor = row.cursor + if (!send('fact', row, cursor)) { + await once(res, 'drain', { signal }) + backpressured = true + break // Reauthenticate/requery instead of draining a stale buffered batch. + } + } + if (!backpressured && !page.events.length && !res.destroyed && !res.write(': keep-alive\n\n')) await once(res, 'drain', { signal }) + page = null + if (!backpressured) await delay(pollMs, undefined, { signal }) + } + } catch (error) { + if (!signal.aborted) { + const code = typeof error.code === 'string' && error.code.startsWith('bot/') ? error.code : 'bot/unavailable' + const details = { nextAction: ['bot/cursor-expired', 'bot/conflict'].includes(code) ? 'project.snapshot' : ['bot/unauthorized', 'bot/forbidden'].includes(code) ? 'authenticate' : 'retry' } + if (!res.headersSent) respond(code === 'bot/unauthorized' ? 401 : code === 'bot/forbidden' ? 403 : ['bot/cursor-expired', 'bot/conflict'].includes(code) ? 409 : code === 'bot/invalid' ? 400 : 503, { ok: false, error: { code, message: 'Project event request could not continue', details } }) + else send(['bot/unauthorized', 'bot/forbidden'].includes(code) ? 'access-revoked' : 'reset-required', { code, details }) + } + } finally { + controller.abort(); active.delete(controller); res.removeListener('close', close) + if (!res.writableEnded && !res.destroyed) res.end() + } + } + return { handle, close() { for (const controller of active) controller.abort() }, active: () => active.size } +} diff --git a/patches/ds-bot/files/src/host/project-service-views.js b/patches/ds-bot/files/src/host/project-service-views.js new file mode 100644 index 00000000..510acf31 --- /dev/null +++ b/patches/ds-bot/files/src/host/project-service-views.js @@ -0,0 +1,93 @@ +import { nativeExitConfirmed } from './project-native-sessions.js' +import { confirmedWorkerExit } from './project-task-sessions.js' +// Public projections retain references to exact scoped sources; never copy an +// entire growing lifecycle or caller authentication context into every list row. +const pick = (row, keys) => Object.fromEntries(keys.filter(key => row?.[key] !== undefined).map(key => [key, structuredClone(row[key])])) +const short = (value, max = 2000) => typeof value === 'string' ? value.slice(0, max) : value +const reference = (taskId, section, count) => ({ query: 'task.history', taskId, section, ...(count === undefined ? {} : { count }) }) +export function taskProvenance(task) { + const source = task.commandSource?.origin + const agentId = source?.agentId ?? task.personalAgentSource?.agentId + return { kind: agentId ? 'personal-agent' : source?.kind ?? 'human', memberId: task.requesterId ?? null, ...(agentId ? { agentId } : {}), ...(source?.botId ? { botId: source.botId } : {}), ...(task.personalAgentSource?.approvedBy ? { approvedBy: task.personalAgentSource.approvedBy } : {}), scope: 'project-published' } +} +export function taskContractView(task) { + const row = task.projectTask + if (!row) return null + return { version: row.version, projectionVersion: 'project-task-contract-summary/v1', admission: { ...pick(row.admission, ['status', 'revision']), latestDecision: row.admission.decisions?.at(-1) ? pick(row.admission.decisions.at(-1), ['id', 'runId', 'at']) : null }, + requirements: { ...pick(row.requirements, ['revision', 'updatedAt']), ...Object.fromEntries(['title', 'criteria', 'impact', 'scope'].filter(key => row.requirements[key] !== undefined).map(key => [key, short(row.requirements[key])])) }, + parentTaskId: row.parentTaskId ?? null, + wait: row.wait ? { ...pick(row.wait, ['id', 'status', 'botId', 'claimId', 'resumedClaimId', 'at', 'resumedAt']), resumePoint: short(row.wait.resumePoint) } : null, + dependencies: (row.dependencies ?? []).slice(0, 16).map(dependency => ({ taskId: dependency.taskId, status: dependency.status, + result: dependency.result ? pick(dependency.result, ['kind', 'taskId', 'sourceTaskId', 'replacementTaskId', 'replacementVersion', 'decisionId', 'attemptId', 'candidateSha', 'requirementsRevision', 'integrated']) : null })), + block: row.block ? { ...pick(row.block, ['kind', 'attemptId', 'at']), reason: short(row.block.reason) } : null, + historyRefs: Object.fromEntries(['requirementsHistory', 'waitHistory', 'results', 'admissionDecisions'].map(section => [section, reference(task.id, section, section === 'admissionDecisions' ? row.admission.decisions?.length ?? 0 : row[section]?.length ?? 0)])), + currentRef: reference(task.id, 'requirements'), preview: true } +} +export function taskDetailView(task) { + const result = pick(task, ['id', 'projectId', 'requesterId', 'projectTaskKind', 'status', 'priority', 'progress', 'createdAt', 'updatedAt', 'waitingFor', 'approvalId', 'inputApprovalIds']) + for (const key of ['title', 'criteria', 'note']) if (task[key] !== undefined) result[key] = short(task[key]) + if (task.target) result.target = { ...pick(task.target, ['owner', 'repo', 'number']), href: short(task.target.href) } + if (task.inputRequest) result.inputRequest = { ...pick(task.inputRequest, ['id', 'status']), question: short(task.inputRequest.question) } + if (task.result) result.result = { summary: short(task.result.summary), ...pick(task.result, ['at', 'sha', 'attemptId']) } + result.projectionVersion = 'project-task-detail/v1' + result.source = taskProvenance(task) + result.projectTask = taskContractView(task) + result.historyRefs = Object.fromEntries(['note', 'result', 'actors', 'handoffs'].map(section => [section, reference(task.id, section)])) + return result +} +export function taskHistorySource(task, section) { + if (section === 'admissionDecisions') return task.projectTask?.admission.decisions ?? [] + if (['requirements', 'requirementsHistory', 'waitHistory', 'results', 'dependencies'].includes(section)) return task.projectTask?.[section] ?? (section === 'requirements' ? null : []) + if (section === 'handoffs') return task.collaboration?.history ?? [] + if (['note', 'result', 'actors'].includes(section)) return task[section] ?? null + throw Object.assign(new Error('Unknown scoped Task history section'), { code: 'bot/invalid' }) +} +export function sourcePage(value, body, identity) { + const text = JSON.stringify(value), offset = body.offset ?? 0, limit = body.limit ?? 8000 + if (!Number.isSafeInteger(offset) || offset < 0 || offset > text.length || !Number.isSafeInteger(limit) || limit < 1 || limit > 16000) throw Object.assign(new Error('Invalid source page'), { code: 'bot/invalid' }) + const sha256 = identity(text) + if (body.expectedSha256 !== undefined && body.expectedSha256 !== sha256) throw Object.assign(new Error('Source changed; restart from its current reference'), { code: 'bot/conflict' }) + const next = Math.min(text.length, offset + limit) + return { sha256, offset, text: text.slice(offset, next), totalChars: text.length, nextOffset: next < text.length ? next : null } +} + +export function attemptMetadataView(attempt) { + const result = pick(attempt, ['id', 'protocol', 'projectId', 'taskId', 'workflowId', 'workflowVersion', 'runId', 'status', 'initiatedBy', 'baseSha', 'seedSha', 'candidateSha', 'branch', 'createdAt', 'updatedAt', 'acceptedAt', 'acceptedBy', 'acceptedViaChief', 'cancelRequested', 'supersededBy', 'roleAssignments']) + if (attempt.requirement) result.requirement = { ...pick(attempt.requirement, ['repository', 'projectTaskRevision']), ...Object.fromEntries(['title', 'criteria', 'impact', 'scope', 'note'].filter(key => attempt.requirement[key] !== undefined).map(key => [key, short(attempt.requirement[key])])) } + if (attempt.failure) result.failure = { ...pick(attempt.failure, ['at']), message: short(attempt.failure.message) } + if (attempt.feedback) result.feedback = { ...pick(attempt.feedback, ['previousAttemptId', 'kind', 'candidateSha']), reason: short(attempt.feedback.reason) } + if (attempt.pr) result.pr = pick(attempt.pr, ['number', 'sha', 'url', 'repository']) + if (attempt.publicationBinding) result.publicationBinding = pick(attempt.publicationBinding, ['number', 'repository', 'branch', 'previousSha', 'upstreamSha', 'upstreamRef']) + if (attempt.integrationInputs) result.integrationInputs = attempt.integrationInputs.slice(0, 20).map(row => pick(row, ['version', 'repository', 'ref', 'sha'])) + if (attempt.workspaces) result.workspaces = Object.fromEntries(Object.entries(attempt.workspaces).slice(0, 32).map(([role, row]) => [role, pick(row, ['id', 'status', 'role', 'resourceId'])])) + if (attempt.agentDefinitions) result.agentDefinitions = { revision: attempt.agentDefinitions.revision, roles: (attempt.agentDefinitions.roles ?? []).slice(0, 32).map(row => ({ ...pick(row, ['id', 'role', 'policyRole', 'name', 'model']), skills: (row.skills ?? []).slice(0, 20).map(value => short(value, 300)), execution: pick(row.execution, ['harness', 'model', 'resourceIds']) })) } + function evidence(row) { return { ...pick(row, ['sha', 'sha256', 'passed', 'verdict', 'harness', 'sdkVersion', 'resourceId', 'sessionId', 'model', 'startedAt', 'completedAt']), ...(row.summary ? { summary: short(row.summary) } : {}) } } + result.evidence = {} + for (const [key, row] of Object.entries(attempt.evidence ?? {}).slice(0, 32)) { + if (key === 'execution') result.evidence.execution = Object.fromEntries(Object.entries(row).slice(0, 32).map(([role, value]) => [role, evidence(value)])) + else result.evidence[key] = evidence(row) + } + result.sourceRefs = Object.fromEntries(['evidence', 'feedback', 'requirement', 'failure', 'agentDefinitions', 'integrationAdmissions'].map(section => [section, { query: 'attempt.context', attemptId: attempt.id, section }])) + result.projectionVersion = 'project-attempt-summary/v1' + result.preview = true + return result +} + +// A scoped physical execution fact, not an inference from a missing Bot or PID. +export function executionMetadataView(state, row) { + if (!row) return null + const native = Object.hasOwn(state.projectNativeWorkers ?? {}, row.id) + const attempt = state.projectAttempts?.[row.attemptId] + const session = [attempt?.roleSessions?.[row.role], ...(attempt?.roleSessionHistory ?? [])].find(value => value?.id === row.sessionKey) + const lease = row.leaseId && state.projectResourceLeases?.[row.leaseId] + const exit = native ? nativeExitConfirmed(state, row) : confirmedWorkerExit(state, row) + return { ...pick(row, ['id', 'role', 'resourceId', 'generation', 'sessionKey', 'logicalSessionId', 'createdAt', 'executionStartedAt', 'releasedAt']), + harness: native ? 'flybot' : row.harness ?? null, status: row.status, + policyRole: session?.policyRole ?? row.policyRole ?? row.role ?? null, + definitionRevision: session?.definitionRevision ?? null, + activationEffectId: native ? row.activationEffect?.id ?? null : null, + exitConfirmed: Boolean(exit && !state.bots?.[row.id] && (!row.leaseId || lease?.status === 'released')), + exitProof: row.releaseProof ? pick(row.releaseProof, ['kind', 'generation', 'at', 'registeredWorkEmpty']) : null, + pending: row.pending ? pick(row.pending, ['id', 'kind']) : null, + ...(row.pendingWork ? { pendingWork: row.pendingWork.slice(0, 32).map(value => ({ kind: typeof value.kind === 'string' ? value.kind.slice(0, 80) : 'unknown', count: Number.isSafeInteger(value.count) ? value.count : 0 })) } : {}) } +} diff --git a/patches/ds-bot/files/src/host/tasks.js b/patches/ds-bot/files/src/host/tasks.js index 6c4af1bf..3c3fe7f1 100644 --- a/patches/ds-bot/files/src/host/tasks.js +++ b/patches/ds-bot/files/src/host/tasks.js @@ -1458,6 +1458,8 @@ export function install(rt) { rt.projects?.stampTask(task, body) if (actor?.personalAgentCredentialId) task.personalAgentSource = rt.personalAgentAuth.source(actor, task.projectId) rt.projects?.validateTask(task, body) + const commandSource = task.projectId && rt.projectDevelopment?.commandSource(task.projectId) + if (commandSource) task.commandSource = commandSource if (task.projectId) { const kind = body.projectTaskKind ?? 'code' if (!['code', 'decision', 'release'].includes(kind)) throw invalid('Unknown Project Task kind') diff --git a/patches/ds-bot/files/src/host/workbench-gateway.js b/patches/ds-bot/files/src/host/workbench-gateway.js index 588c58bc..7c0e794b 100644 --- a/patches/ds-bot/files/src/host/workbench-gateway.js +++ b/patches/ds-bot/files/src/host/workbench-gateway.js @@ -1,6 +1,7 @@ // Restricted shared entry. The DSH owner credential stays server-side; members never // receive raw Harness APIs or its owner cookie. Project authorization remains in FlyBot. import { createServer } from 'node:http' +import { createProjectSse } from './project-service-sse.js' import { gzipSync } from 'node:zlib' import { readFile, stat } from 'node:fs/promises' import { resolve } from 'node:path' @@ -25,13 +26,25 @@ export const WORKBENCH_ENDPOINTS = new Set([ const LIMIT = 1024 * 1024 const HTML = 'FlyBot · Project workbench
' -export function createWorkbenchGateway({ upstreamOrigin, upstreamCookie, publicOrigin, readAsset, request = fetch } = {}) { +export function createWorkbenchGateway({ upstreamOrigin, upstreamCookie, publicOrigin, readAsset, request = fetch, eventPollMs = 1000, eventMaxStreams = 64 } = {}) { const upstream = new URL(upstreamOrigin) if (!['http:', 'https:'].includes(upstream.protocol) || !['localhost', '127.0.0.1', '[::1]'].includes(upstream.hostname) || upstream.pathname !== '/' || upstream.search || upstream.hash || upstream.username || upstream.password) throw new Error('Workbench upstream must be a fixed loopback origin') if (typeof upstreamCookie !== 'string' || !upstreamCookie || upstreamCookie.length > 8192 || /[\r\n\u0000]/.test(upstreamCookie)) throw new Error('A valid private upstream cookie is required') const published = publicOrigin ? new URL(publicOrigin) : null if (published && (published.origin !== publicOrigin || published.username || published.password || !['http:', 'https:'].includes(published.protocol))) throw new Error('Invalid fixed workbench public origin') const assets = readAsset ?? (name => readFile(resolve(import.meta.dirname, '../..', name))) + const events = createProjectSse({ pollMs: eventPollMs, maxStreams: eventMaxStreams, + async pull({ projectId, name, payload, token, signal }) { + const response = await request(`${upstream.origin}/api/bot`, { method: 'POST', redirect: 'error', signal: AbortSignal.any([signal, AbortSignal.timeout(10000)]), + headers: { 'content-type': 'application/json', cookie: upstreamCookie, origin: upstream.origin, 'x-flybot-member-session': token }, + body: JSON.stringify({ endpoint: 'project-development', payload: { version: 'project-development/v1', kind: 'query', name, projectId, payload } }) }) + if (!response.ok) throw new Error('Project event runtime is unavailable') + const text = await response.text() + if (text.length > LIMIT) throw new Error('Project event page exceeded its limit') + const result = JSON.parse(text) + if (!result.ok) throw Object.assign(new Error('Project event access/query refused'), { code: result.error?.code }) + return result.value.result + } }) const server = createServer(async (req, res) => { res.setHeader('cache-control', 'no-store') res.setHeader('x-content-type-options', 'nosniff') @@ -53,6 +66,7 @@ export function createWorkbenchGateway({ upstreamOrigin, upstreamCookie, publicO const origin = req.headers.origin if (origin && !allowed.includes(origin) || req.headers['sec-fetch-site'] === 'cross-site') return respond(403, { error: 'Cross-origin request refused' }) const url = new URL(req.url ?? '/', allowed[0]) + if (req.method === 'GET' && url.pathname === '/api/project-events') return events.handle(req, res, url, respond) if (url.search) return respond(404, { error: 'Not found' }) if (req.method === 'GET') { if (url.pathname === '/') return respond(200, HTML, 'text/html; charset=utf-8') @@ -91,6 +105,8 @@ export function createWorkbenchGateway({ upstreamOrigin, upstreamCookie, publicO return respond(200, result) // Never forward upstream Set-Cookie/Location/owner credentials. } catch { if (!res.headersSent) respond(502, { error: 'Workbench runtime request failed' }) } }) + const close = server.close + server.close = function (...args) { events.close(); return close.apply(this, args) } server.on('upgrade', (_request, socket) => { socket.end('HTTP/1.1 403 Forbidden\r\nConnection: close\r\n\r\n') }) return server } @@ -99,7 +115,7 @@ if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1] const value = name => { const index = process.argv.indexOf(name); return index < 0 ? undefined : process.argv[index + 1] } const file = value('--upstream-cookie-file') if (!file || (await stat(file)).mode & 0o077) throw new Error('Use a private mode-0600 upstream cookie file') - const server = createWorkbenchGateway({ upstreamOrigin: value('--upstream'), upstreamCookie: (await readFile(file, 'utf8')).trim(), publicOrigin: value('--public-origin') }) + const server = createWorkbenchGateway({ upstreamOrigin: value('--upstream'), upstreamCookie: (await readFile(file, 'utf8')).trim(), publicOrigin: value('--public-origin'), eventPollMs: Number(value('--event-poll-ms') ?? 1000), eventMaxStreams: Number(value('--event-max-streams') ?? 64) }) const port = Number(value('--port') ?? 3081) server.listen(port, '127.0.0.1', () => process.stdout.write(`Project workbench: http://127.0.0.1:${server.address().port}/\n`)) for (const signal of ['SIGTERM', 'SIGINT']) process.once(signal, () => server.close()) diff --git a/patches/ds-bot/files/test/project-agents.test.js b/patches/ds-bot/files/test/project-agents.test.js index 464ae7fb..319a4c48 100644 --- a/patches/ds-bot/files/test/project-agents.test.js +++ b/patches/ds-bot/files/test/project-agents.test.js @@ -879,3 +879,33 @@ test('role definitions keep exact instruction limits and reject capability/forei await assert.rejects(rt.projectEndpoint('project-agent-definition', { projectId: 'p1', role: 'chief', policyRole: 'developer' }, alice), /capability class/) assert.equal(rt.projectAgents.deliveryDefinitions('p1').revision, revision) }) + +test('versioned role queries and Owner approval reuse original governance and receipt replay', async t => { + const { rt } = await setup(t), initial = await get(rt, alice) + await message(rt, bob, 'public', 'Suggest an analysis role through the service') + const chief = rt.botOf(initial.public.botId) + const proposed = await rt.projectAgents.proposeRole(chief, { requestId: 'v1-analysis-role', action: 'add', role: 'analyst', reason: 'Independent technical analysis supports Project decisions', fields: { name: 'Analyst', policyRole: 'reviewer', instructions: 'Follow the repository Review conventions', skills: ['commit-review'] } }) + const query = { version: 'project-development/v1', kind: 'query', name: 'role.proposals', projectId: 'p1', payload: {} } + const list = await rt.projectEndpoint('project-development', query, bob) + assert.ok(list.result.proposals.some(row => row.id === proposed.proposal.id)) + const approval = { version: 'project-development/v1', kind: 'command', name: 'role.decide', projectId: 'p1', requestId: 'v1-approve-analysis', payload: { proposalId: proposed.proposal.id, expectedSha256: proposed.proposal.sha256, decision: 'approve' } } + await assert.rejects(rt.projectEndpoint('project-development', approval, bob), /access denied|denied/i) + const first = await rt.projectEndpoint('project-development', approval, alice) + assert.equal(first.result.proposal.status, 'approved') + const revision = rt.projectAgents.deliveryDefinitions('p1').revision + const replay = await rt.projectEndpoint('project-development', approval, alice) + assert.equal(replay.replayed, true) + assert.equal(rt.projectAgents.deliveryDefinitions('p1').revision, revision) + const roles = await rt.projectEndpoint('project-development', { ...query, name: 'roles' }, bob) + assert.ok(roles.result.roles.some(row => row.role === 'analyst' && row.policyRole === 'reviewer')) +}) + +test('reading an implicit legacy role catalog does not materialize or mutate Project facts', async t => { + const { rt } = await setup(t) + if (rt.state.projectAgentDefinitions) delete rt.state.projectAgentDefinitions.p1 + const before = JSON.stringify(rt.state.projectAgentDefinitions ?? {}) + const response = await rt.projectEndpoint('project-development', { version: 'project-development/v1', kind: 'query', name: 'roles', projectId: 'p1', payload: {} }, bob) + assert.equal(response.result.revision, 1) + assert.equal(response.result.roles.length, 4) + assert.equal(JSON.stringify(rt.state.projectAgentDefinitions ?? {}), before) +}) diff --git a/patches/ds-bot/files/test/project-command-receipts.test.js b/patches/ds-bot/files/test/project-command-receipts.test.js new file mode 100644 index 00000000..2135e235 --- /dev/null +++ b/patches/ds-bot/files/test/project-command-receipts.test.js @@ -0,0 +1,140 @@ +import assert from 'node:assert/strict' +import { test } from 'node:test' +import { createProjectEffects } from '../src/host/project-effects.js' +import { createCommandReceipts } from '../src/host/project-command-receipts.js' +function fixture(initial) { + const state = structuredClone(initial ?? { members: { alice: {}, bob: {} }, tasks: { t: { id: 't', projectId: 'p', requesterId: 'alice' } } }) + let disk, failSave = false, saves = 0, revoke = false + const rt = { state, readOnly: () => undefined, + requireProjectAccess(actor, projectId, action = 'read') { if (revoke || projectId !== 'p' || !state.members[actor?.memberId] || action !== 'read' && actor.memberId !== 'alice') throw Object.assign(Error('Current access denied'), { code: 'bot/forbidden' }) }, + async save() { ++saves; if (failSave) throw Error('fixture durable failure'); const capture = rt.projectEffects.capture(); disk = structuredClone(state); rt.projectEffects.saved(capture) }, + } + rt.projectEffects = createProjectEffects(rt) + const receipts = createCommandReceipts(rt), actor = { memberId: 'alice' } + const spec = { projectId: 'p', requestId: 'once', name: 'fixture.command', args: { taskId: 't' }, action: 'task-write', taskId: 't', actor } + return { rt, receipts, spec, actor, disk: () => disk, fail: value => { failSave = value }, revoke: () => { revoke = true }, saves: () => saves } +} + +test('concurrent commands commit one result; retries ignore advanced mutable preconditions and reject intent conflicts', async () => { + const f = fixture(); let invoked = 0, preconditions = 0 + const invoke = async () => ({ changed: ++invoked }), check = () => { if (++preconditions > 2) throw Error('stale generation') } + const [first, second] = await Promise.all([f.receipts.run(f.spec, invoke, check), f.receipts.run(f.spec, invoke, check)]) + assert.equal(invoked, 1); assert.deepEqual(first.result, second.result) + const retry = await f.receipts.run(f.spec, invoke, () => { throw Error('advanced generation must not reject a confirmed replay') }) + assert.equal(retry.replayed, true); assert.equal(invoked, 1) + assert.equal(f.rt.projectEffects.intentCommitted(f.rt.state.projectCommandReceipts[first.receipt.id].resultEffect), true) + await assert.rejects(f.receipts.run({ ...f.spec, args: { taskId: 'different' } }, invoke), error => error.code === 'bot/idempotency-conflict') + f.revoke(); await assert.rejects(f.receipts.run(f.spec, invoke), /access denied/) + await f.rt.projectEffects.close() +}) + +test('failed intent cannot dispatch; lost response recovers its durable result after restart without invoking again', async () => { + const f = fixture(); let invoked = 0 + f.fail(true) + await assert.rejects(f.receipts.run(f.spec, async () => ({ changed: ++invoked })), /durable failure/) + assert.equal(invoked, 0); assert.equal(Object.keys(f.rt.state.projectCommandReceipts).length, 0) + f.fail(false) + await f.receipts.run(f.spec, async () => ({ changed: ++invoked })) + const g = fixture(f.disk()); g.rt.projectEffects.recover() + const recovered = await g.receipts.run(g.spec, async () => { throw Error('Must not execute') }) + assert.deepEqual(recovered.result, { changed: 1 }); assert.equal(recovered.replayed, true) + assert.equal(invoked, 1) + await Promise.all([f.rt.projectEffects.close(), g.rt.projectEffects.close()]) +}) + +test('ambiguous command outcomes do not silently repeat and private receipt/effect access stays principal scoped', async () => { + const f = fixture(); let invoked = 0 + await assert.rejects(f.receipts.run(f.spec, async () => { ++invoked; throw Error('External outcome unknown') }), /outcome unknown/) + await assert.rejects(f.receipts.run(f.spec, async () => { ++invoked }), error => error.code === 'bot/outcome-unknown') + assert.equal(invoked, 1) + assert.equal(f.receipts.get('p', 'once', f.actor).status, 'unknown') + assert.throws(() => f.receipts.get('p', 'once', { memberId: 'bob' }), /unavailable/) + assert.equal(f.rt.projectEffects.list('p', { memberId: 'bob' }).length, 0) + const own = f.rt.projectEffects.list('p', f.actor) + assert.ok(own.length >= 2) + await assert.rejects(f.rt.projectEffects.retry({ projectId: 'p', effectId: own[0].id, requestId: 'unauthorized' }, { memberId: 'bob' })) + await f.rt.projectEffects.close() +}) + +test('explicit reconciliation seals a result without a second generic invocation; current Task source is stamped once', async () => { + const f = fixture(); let invoked = 0 + const spec = { ...f.spec, name: 'task.propose' } + await assert.rejects(f.receipts.run(spec, async () => { + ++invoked + const source = f.receipts.source('p') + assert.equal(source.origin.kind, 'human'); assert.ok(source.commandReceiptId) + assert.equal(f.receipts.source('p'), null) + throw Error('response lost after target commit') + })) + const recovered = await f.receipts.run({ ...spec, reconcile: async () => ({ result: { taskId: 'durable-target' } }) }, async () => { ++invoked }) + assert.equal(invoked, 1); assert.equal(recovered.reconstructed, true) + const page = f.receipts.result('p', { commandRequestId: 'once', limit: 1000 }, f.actor) + assert.equal(page.text, JSON.stringify({ taskId: 'durable-target' })) + assert.throws(() => f.receipts.result('p', { commandRequestId: 'once', expectedSha256: 'stale' }, f.actor), /changed/) + await f.rt.projectEffects.close() +}) + +test('an observed result without its final commit is labelled pending and retries finish it without executing twice', async () => { + const f = fixture(); let invoked = 0 + await assert.rejects(f.receipts.run(f.spec, async () => { ++invoked; f.fail(true); return { changed: true } }), /durable failure/) + const pending = f.receipts.get('p', 'once', f.actor) + assert.equal(pending.status, 'completion-pending-commit'); assert.equal(pending.committed, false) + assert.throws(() => f.receipts.result('p', { commandRequestId: 'once' }, f.actor), /not committed/) + f.fail(false) + const recovered = await f.receipts.run(f.spec, async () => { ++invoked }) + assert.equal(invoked, 1); assert.equal(recovered.receipt.committed, true) + assert.deepEqual(recovered.result, { changed: true }) + await f.rt.projectEffects.close() +}) + +test('large recovered results are hash scoped, paged and refuse symlinked payloads', async t => { + const fs = await import('node:fs'), { tmpdir } = await import('node:os'), { join } = await import('node:path') + const f = fixture(); f.rt.home = fs.mkdtempSync(join(tmpdir(), 'flybot-command-result-')) + t.after(async () => { await f.rt.projectEffects.close(); fs.rmSync(f.rt.home, { recursive: true, force: true }) }) + const result = { text: 'result data '.repeat(10000) } + const completed = await f.receipts.run(f.spec, async () => result) + assert.equal((await f.receipts.run(f.spec, async () => { throw Error('No redispatch') })).result.text, result.text) + const row = f.rt.state.projectCommandReceipts[completed.receipt.id] + assert.equal(row.resultText, undefined); assert.equal(row.resultFile, true) + assert.equal(f.receipts.result('p', { commandRequestId: 'once', limit: 99 }, f.actor).text.length, 99) + const file = join(f.rt.home, 'project-command-results', row.resultSha256 + '.json'), outside = join(f.rt.home, 'outside-reference.json') + fs.writeFileSync(outside, JSON.stringify(result)); fs.unlinkSync(file); fs.symlinkSync(outside, file) + assert.throws(() => f.receipts.result('p', { commandRequestId: 'once' }, f.actor), /regular file/) +}) + +test('an omitted candidate precondition differs from explicitly requiring no candidate', async () => { + const f = fixture() + await f.receipts.run(f.spec, async () => ({ changed: true })) + await assert.rejects(f.receipts.run({ ...f.spec, expectedCandidateSha: null }, async () => ({ changed: false })), error => error.code === 'bot/idempotency-conflict') + await f.rt.projectEffects.close() +}) + +test('native and external connections of one independent personal agent share receipts across carrier changes', async () => { + const f = fixture() + f.rt.state.projectAgents = { carrier: { personalAgentId: 'independent-agent' }, replacement: { personalAgentId: 'independent-agent' } } + const native = { memberId: 'alice', agentId: 'carrier', watchGrantId: 'grant-1' } + const external = { memberId: 'alice', agentId: 'independent-agent', personalAgentCredentialId: 'external-credential' } + let invoked = 0 + const first = await f.receipts.run({ ...f.spec, actor: native }, async () => ({ changed: ++invoked })) + assert.equal(first.receipt.origin.kind, 'personal-agent') + assert.equal(first.receipt.origin.agentId, 'independent-agent') + const repeated = await f.receipts.run({ ...f.spec, actor: external }, async () => ({ changed: ++invoked })) + const rotated = await f.receipts.run({ ...f.spec, actor: { ...native, agentId: 'replacement', watchGrantId: 'grant-2' } }, async () => ({ changed: ++invoked })) + assert.equal(invoked, 1); assert.equal(first.receipt.id, repeated.receipt.id); assert.equal(rotated.receipt.id, first.receipt.id) + f.rt.state.members.alice.status = 'disabled' + await assert.rejects(f.receipts.run({ ...f.spec, actor: external }, async () => ({})), /disabled/) + await f.rt.projectEffects.close() +}) + +test('request IDs are operation scoped; ambiguous lookups require a name and missing receipt queries are pure', async () => { + const f = fixture(), original = JSON.stringify(f.rt.state) + assert.throws(() => f.receipts.get('p', 'missing', f.actor), /unavailable/) + assert.equal(JSON.stringify(f.rt.state), original) + const first = await f.receipts.run(f.spec, async () => ({ first: true })) + const second = await f.receipts.run({ ...f.spec, name: 'fixture.other' }, async () => ({ second: true })) + assert.notEqual(first.receipt.id, second.receipt.id) + assert.throws(() => f.receipts.get('p', 'once', f.actor), error => error.code === 'bot/ambiguous-command') + assert.equal(f.receipts.get('p', 'once', f.actor, 'fixture.other').id, second.receipt.id) + assert.equal(f.receipts.result('p', { commandRequestId: 'once', commandName: f.spec.name }, f.actor).text, '{"first":true}') + await f.rt.projectEffects.close() +}) diff --git a/patches/ds-bot/files/test/project-delivery.test.js b/patches/ds-bot/files/test/project-delivery.test.js index 7f162680..c8fc9656 100644 --- a/patches/ds-bot/files/test/project-delivery.test.js +++ b/patches/ds-bot/files/test/project-delivery.test.js @@ -1185,20 +1185,22 @@ test('an approved extra merge seals when the published upstream is already an an assert.equal(f.attempt.pr,undefined) }) -const integrationCall = (f, kind, name, payload, actor = alice, requestId = 'approve-backend') => f.rt.projectEndpoint('project-development', { version: 'project-development/v1', kind, name, projectId: 'p1', ...(kind === 'command' ? { requestId } : {}), payload: { attemptId: f.attempt.id, ...payload } }, actor) +const integrationCall = (f, kind, name, payload, actor = alice, requestId = `integration-${f.integrationSequence = (f.integrationSequence ?? 0) + 1}`) => f.rt.projectEndpoint('project-development', { version: 'project-development/v1', kind, name, projectId: 'p1', ...(kind === 'command' ? { requestId } : {}), payload: { attemptId: f.attempt.id, ...payload } }, actor) const indexBytes = f => createHash('sha256').update(readFileSync(git(f.path,['rev-parse','--git-path','index']))).digest('hex') test('manager explicitly pins integration input through service snapshot without changing index or approving delivery', async t => { const f=await additionalIntegrationFixture(t),before=indexBytes(f) const snapshot=(await integrationCall(f,'query','attempt.integration-state',{})).result const payload={inputs:[{ref:f.ref,sha:f.source}],generation:snapshot.generation,expectedWorkspace:snapshot.snapshot} - const approved=await integrationCall(f,'command','attempt.integration-bind',payload) + const approved=await integrationCall(f,'command','attempt.integration-bind',payload,alice,'approve-backend') assert.equal(approved.result.already,false) assert.equal(approved.result.admission.generation,snapshot.generation+1) assert.equal(indexBytes(f),before) assert.equal(git(f.path,['rev-parse','MERGE_HEAD']),f.source) const reordered={...payload,expectedWorkspace:Object.fromEntries(Object.entries(payload.expectedWorkspace).reverse())} - assert.equal((await integrationCall(f,'command','attempt.integration-bind',reordered)).result.already,true) + const replay=await integrationCall(f,'command','attempt.integration-bind',reordered,alice,'approve-backend') + assert.equal(replay.replayed,true) + assert.equal(replay.result.admission.id,approved.result.admission.id) assert.equal(f.attempt.integrationAdmissions.length,1) assert.equal(f.attempt.candidateSha,undefined) const node=await f.rt.projectDelivery.preparePoolRole(f.task,{id:'resumed-integration',attemptId:f.attempt.id,generation:99},f.developer) @@ -1335,9 +1337,9 @@ test('approved object availability alone cannot replace a real merge or reuse an const f=await additionalIntegrationFixture(t,{merge:false}) const state=(await integrationCall(f,'query','attempt.integration-state',{})).result const payload={inputs:[{ref:f.ref,sha:f.source}],generation:state.generation,expectedWorkspace:state.snapshot} - await integrationCall(f,'command','attempt.integration-bind',payload) + await integrationCall(f,'command','attempt.integration-bind',payload,alice,'reuse-source') const before=indexBytes(f) - await assert.rejects(integrationCall(f,'command','attempt.integration-bind',{...payload,generation:state.generation+1}),/requestId was reused/) + await assert.rejects(integrationCall(f,'command','attempt.integration-bind',{...payload,generation:state.generation+1},alice,'reuse-source'),/requestId was reused/) await assert.rejects(f.rt.projectDelivery.completePoolRole({task:f.task,claim:{attemptId:f.attempt.id},role:'developer',outputs:{}}),/merge every pinned integration/) assert.equal(indexBytes(f),before) assert.equal(git(f.path,['rev-parse','HEAD']),f.seed) @@ -1531,6 +1533,8 @@ test('a parent cannot seal skipped or subsequently invalidated child integration test('approved custom role capabilities preserve logical identities while sealing testing and reviewing real code', async t => { const f = await setup(t, undefined, { pool: true }) f.rt.projectPools.connect({ validate: () => ({ ok: true }) }) + // Explicit setup is a write; definition metadata reads do not materialize catalogs. + await f.rt.projectEndpoint('project-agent-definition', { projectId: 'p1', role: 'developer', name: 'Developer' }, alice) const original = f.rt.projectAgents.deliveryDefinitions('p1') const names = { developer: 'implementer', tester: 'validator', reviewer: 'auditor' } // Governance approval is tested through real host APIs separately. This fixture diff --git a/patches/ds-bot/files/test/project-development-service.test.js b/patches/ds-bot/files/test/project-development-service.test.js index 0a41059e..28429308 100644 --- a/patches/ds-bot/files/test/project-development-service.test.js +++ b/patches/ds-bot/files/test/project-development-service.test.js @@ -132,7 +132,7 @@ test('trusted role and schedule ports retain original claims, authority and even test('proposal/worker targets stay in envelope scope and permission request identity is preserved', async () => { const { service, state, rt, actor } = fixture() state.projectAgentProposals = { foreign: { projectId: 'other', memberId: 'alice' } } - state.projectHarnessWorkers = { foreign: { projectId: 'other' }, own: { projectId: 'p' } } + state.projectHarnessWorkers = { foreign: { projectId: 'other' }, own: { projectId: 'p', taskId: 't' } } let calls = 0 rt.projectAgents.endpoint = async () => { calls++; return {} } rt.projectHarness = { input: async body => { calls++; assert.equal(body.requestId, 'permission-request'); return { sent: true } } } @@ -213,3 +213,96 @@ test('publication recovery shows its actual wait and stale outbox candidates can task.chiefQuestion = { reason: 'Human scope decision' } assert.equal(taskView(state, task).phase, 'awaiting-owner') }) + +test('Task summaries bound historical context and never expose execution requester sessions', async () => { + const { service, state, actor } = fixture(), task = state.tasks.t + task.title = 'Current requirement' + task.projectTask = { version: 'project-task/v1', admission: { status: 'accepted' }, requirements: { revision: 7, title: task.title, criteria: 'Regression passes', impact: 'Improve the Project', scope: 'Current scope' }, dependencies: [], + requirementsHistory: Array.from({ length: 120 }, (_, n) => ({ revision: n, scope: 'historical scope '.repeat(2000) })), waitHistory: [], results: [], executionRequester: { memberId: 'alice', sessionId: 'private-requester-session' } } + const before = JSON.stringify(state) + const view = service.view(task, actor) + assert.ok(Buffer.byteLength(JSON.stringify(view)) < 16000) + assert.equal(JSON.stringify(view).includes('private-requester-session'), false) + assert.equal(view.contract.requirements.revision, 7) + assert.equal(view.contract.historyRefs.requirementsHistory.count, 120) + const query = { version: DEVELOPMENT_PROTOCOL, kind: 'query', name: 'task.history', projectId: 'p', payload: { taskId: 't', section: 'requirementsHistory', limit: 16000 } } + const first = (await service.endpoint('project-development', query, actor)).result + assert.equal(first.text.length, 16000) + assert.equal(first.text, JSON.stringify(task.projectTask.requirementsHistory).slice(0, 16000)) + await assert.rejects(service.endpoint('project-development', { ...query, payload: { ...query.payload, expectedSha256: 'stale' } }, actor), /changed/) + await assert.rejects(service.endpoint('project-development', { ...query, payload: { ...query.payload, section: 'executionRequester' } }, actor)) + await assert.rejects(service.endpoint('project-development', { ...query, projectId: 'other' }, actor)) + assert.equal(JSON.stringify(state), before) +}) + +test('Attempt summaries bound repeated sessions, role definitions and evidence while exact scoped sources remain readable', async () => { + const { service, state, actor } = fixture(), huge = 'long role report '.repeat(10000) + state.projectAttempts.large = { id: 'large', projectId: 'p', taskId: 't', createdAt: 1, status: 'failed', failure: { message: huge }, feedback: { reason: huge }, requirement: { title: huge, criteria: huge }, + roleSessionHistory: Array.from({ length: 200 }, (_, n) => ({ id: 'session-' + n, role: 'developer', context: { text: huge, sha256: 'hash' }, policy: { text: huge, sha256: 'policy' } })), + agentDefinitions: { roles: [{ id: 'p:developer', role: 'developer', persona: huge, memory: huge, execution: { harness: 'codex' } }] }, evidence: { review: { sha: 'commit', verdict: 'approve', summary: huge } } } + const before = JSON.stringify(state) + const detail = await service.endpoint('project-task-detail', { taskId: 't' }, actor) + assert.ok(Buffer.byteLength(JSON.stringify(detail)) < 50000) + const summary = detail.attempts[0] + assert.equal(summary.roleSessionHistory.length, 3) + assert.equal(summary.roleSessionHistoryPage.total, 200) + assert.equal(summary.agentDefinitions.roles[0].persona, undefined) + const query = { version: DEVELOPMENT_PROTOCOL, kind: 'query', name: 'attempt.context', projectId: 'p', payload: { attemptId: 'large', section: 'evidence', limit: 16000 } } + assert.equal((await service.endpoint('project-development', query, actor)).result.text, JSON.stringify(state.projectAttempts.large.evidence).slice(0, 16000)) + assert.equal(JSON.stringify(state), before) +}) + +test('durable v1 command replays its confirmed response after generation advances and keeps source keys distinct', async () => { + const { createProjectEffects } = await import('../src/host/project-effects.js') + const { rt, state, actor } = fixture() + state.tasks.t.collaboration = { generation: 4, history: [] } + rt.projectEffects = createProjectEffects(rt) + rt.save = async () => rt.projectEffects.saved(rt.projectEffects.capture()) + let starts = 0 + rt.projectScheduler.request = async () => { ++starts; state.tasks.t.collaboration.generation = 5; return { queue: { id: 'one-queue', projectId: 'p' } } } + const service = createDevelopmentService(rt) + const request = { version: DEVELOPMENT_PROTOCOL, kind: 'command', name: 'task.run', projectId: 'p', requestId: 'stable-command', expectedGeneration: 4, payload: { taskId: 't' } } + const first = await service.endpoint('project-development', request, actor) + const repeated = await service.endpoint('project-development', request, actor) + assert.equal(starts, 1) + assert.equal(first.receipt.committed, true); assert.equal(repeated.replayed, true) + assert.deepEqual(first.result, repeated.result) + await assert.rejects(service.endpoint('project-development', { ...request, expectedGeneration: 5 }, actor), error => error.code === 'bot/idempotency-conflict') + const query = { version: DEVELOPMENT_PROTOCOL, kind: 'query', name: 'command.receipt', projectId: 'p', payload: { commandRequestId: 'stable-command' } } + assert.equal((await service.endpoint('project-development', query, actor)).result.id, first.receipt.id) + await assert.rejects(service.endpoint('project-development', query, { memberId: 'bob' }), /unavailable/) + await rt.projectEffects.close() +}) + +test('native execution views distinguish committed activation, current roles and confirmed release without host internals', () => { + const { state } = fixture(), task = state.tasks.t + task.deliveryAttemptId = 'a'; task.collaboration = { status: 'claimed', botId: 'p:implementer', generation: 2, history: [], claim: { id: 'claim', workerId: 'w', role: 'implementer', policyRole: 'developer', generation: 2, status: 'active' } } + state.projectAgentDefinitions = { p: { revision: 2, roles: [] } } + state.projectAttempts.a = { id: 'a', projectId: 'p', taskId: 't', status: 'running', roleSessions: { implementer: { id: 'logical', definitionRevision: 1 } } } + state.bots.w = { id: 'w', projectId: 'p', attemptId: 'a' } + const worker = { id: 'w', projectId: 'p', taskId: 't', attemptId: 'a', role: 'implementer', sessionKey: 'logical', generation: 2, version: 'project-native-worker/v2', status: 'bound', sessionId: 'physical', bindingKey: 'binding', activationEffect: { id: 'activate' }, processScope: { unit: 'HIDDEN_HOST_DETAIL' }, failure: 'HIDDEN_HOST_DETAIL' } + state.projectNativeWorkers = { w: worker } + const view = options => taskView(state, task, { activity: { w: 'thinking' }, ...options }) + assert.equal(view({ isIntentCommitted: () => false }).phase, 'starting') + assert.equal(view({ isIntentCommitted: () => true }).phase, 'running') + assert.equal(view().currentRole, 'implementer') + assert.equal(view().roleDefinitionRevision, 2) + assert.equal(view().execution.definitionRevision, 1) + assert.equal(JSON.stringify(view()).includes('HIDDEN_HOST_DETAIL'), false) + worker.status = 'stop-unconfirmed' + assert.equal(view().waitingFor, 'worker-exit') + assert.equal(view().execution.exitConfirmed, false) + worker.status = 'quiescent'; worker.releaseProof = { kind: 'native-agent-quiescent', registeredWorkEmpty: true, workerId: 'w', sessionId: 'physical', bindingKey: 'binding', generation: 2 } + delete state.bots.w + assert.equal(view().execution.exitConfirmed, true) +}) + +test('attempt lists reference large frozen definitions instead of copying their content into each session', async () => { + const { service, rt, state, actor } = fixture(), text = 'frozen persona source '.repeat(20000) + state.projectAttempts.a = { id: 'a', projectId: 'p', taskId: 't', roleSessions: { developer: { id: 'session', role: 'developer', definitionRef: 'original-definition', definitionRevision: 1, definition: { instructions: text, resolvedSkills: [{ content: text }] } } } } + rt.projectDelivery = { endpoint: async () => ({ task: state.tasks.t, attempts: [state.projectAttempts.a] }) } + const result = await service.endpoint('project-task-detail', { taskId: 't' }, actor) + assert.ok(JSON.stringify(result).length < 16000) + assert.equal(result.attempts[0].roleSessions.developer.definitionRevision, 1) + assert.equal(state.projectAttempts.a.roleSessions.developer.definition.instructions, text) +}) diff --git a/patches/ds-bot/files/test/project-effects-runtime.test.js b/patches/ds-bot/files/test/project-effects-runtime.test.js index 8afcc15b..661d72a6 100644 --- a/patches/ds-bot/files/test/project-effects-runtime.test.js +++ b/patches/ds-bot/files/test/project-effects-runtime.test.js @@ -60,7 +60,7 @@ test('v1 Task input commits the inbox and notify intent before returning and cre const disk = JSON.parse(readFileSync(join(f.home, 'state.json'), 'utf8')) assert.equal(disk.projectTaskInputs[first.result.input.id].text, command.payload.text) assert.ok(Object.values(disk.projectEffects).some(row => row.kind === 'input.notify' && row.payload.inputId === first.result.input.id)) - assert.equal((await f.rt.projectEndpoint('project-development', command, { memberId: 'owner-1' })).result.already, true) + assert.equal((await f.rt.projectEndpoint('project-development', command, { memberId: 'owner-1' })).replayed, true) const list = await f.rt.projectEndpoint('project-development', { version: 'project-development/v1', kind: 'query', name: 'inputs', projectId: 'p', payload: { taskId: 'task' } }, { memberId: 'owner-1' }) assert.equal(list.result.inputs.length, 1) assert.equal(Object.keys(f.rt.state.tasks).length, 1) @@ -89,7 +89,9 @@ test('v1 GitHub subscription polling commits source cursor and original Task inp assert.equal(disk.projectThreads[sub.threadId].cursor.pages.comments,1) assert.ok(Object.values(disk.projectEffects).some(row=>row.kind==='input.notify'&&row.payload.inputId===input.id)) assert.equal(Object.keys(disk.tasks).length,1) - assert.equal((await command('source.poll',{threadId:sub.threadId})).result.inputs,0) + assert.equal((await command('source.poll',{threadId:sub.threadId})).replayed,true) + const fresh=await f.rt.projectEndpoint('project-development',{version:'project-development/v1',kind:'command',projectId:'p',name:'source.poll',requestId:'fresh-poll',payload:{threadId:sub.threadId}},actor) + assert.equal(fresh.result.inputs,0) }) test('scheduled Source polling survives closed browser sessions using scoped current-principal broker proof',async t=>{ @@ -119,6 +121,9 @@ test('real membership removal cancels its pending input but preserves the Projec assert.ok(f.rt.state.tasks.task) await assert.rejects(f.rt.projectEndpoint('project-development',command,{memberId:'bob'}),/access denied/) f.rt.state.projects.p.roles.members.push('bob') - assert.equal((await f.rt.projectEndpoint('project-development',command,{memberId:'bob'})).result.already,true) + assert.equal((await f.rt.projectEndpoint('project-development',command,{memberId:'bob'})).replayed,true) assert.equal(f.rt.state.projectTaskInputs[published.result.input.id].status,'cancelled') + const domain=await f.rt.projectInputs.publish({...command.payload,projectId:'p',requestId:command.requestId},{memberId:'bob'}) + assert.equal(domain.already,true) + assert.equal(domain.input.status,'cancelled') }) diff --git a/patches/ds-bot/files/test/project-pools.test.js b/patches/ds-bot/files/test/project-pools.test.js index 81abd880..dfbe700e 100644 --- a/patches/ds-bot/files/test/project-pools.test.js +++ b/patches/ds-bot/files/test/project-pools.test.js @@ -520,6 +520,28 @@ test('a sealed parent resumes through one successor Attempt while retaining its await f.rt.projectEffects.close() }) +test('failed durable acceptance cannot be replayed as a claim; retries commit one active owner', async () => { + const f = await projectTaskFixture() + await f.admit() + const handoffId = f.task.collaboration.currentHandoffId, save = f.rt.save + f.rt.save = async () => { throw new Error('fixture acceptance commit failed') } + await assert.rejects(f.pool.accept('task', 'p:developer', handoffId, 'accept-once'), /commit failed/) + assert.equal(f.task.collaboration.status, 'queued') + assert.equal(f.task.collaboration.claim, undefined) + assert.equal(f.task.collaboration.history.at(-1).acceptance, undefined) + f.rt.save = save + const [first, duplicate] = await Promise.all([ + f.pool.accept('task', 'p:developer', handoffId, 'accept-once'), + f.pool.accept('task', 'p:developer', handoffId, 'accept-once'), + ]) + assert.equal(first.id, duplicate.id) + assert.ok(first.acceptanceEffect) + assert.equal(f.rt.projectEffects.intentCommitted(first.acceptanceEffect), true) + assert.equal(f.task.collaboration.history.at(-1).acceptance.effect.id, first.acceptanceEffect.id) + assert.equal(f.task.collaboration.claim.id, first.id) + await f.rt.projectEffects.close() +}) + test('a retired source can finish its frozen active claim but cannot receive new offers', async () => { const f = setup(); await f.admit(); const claim = await f.claim() diff --git a/patches/ds-bot/files/test/project-service-events.test.js b/patches/ds-bot/files/test/project-service-events.test.js new file mode 100644 index 00000000..ec6de910 --- /dev/null +++ b/patches/ds-bot/files/test/project-service-events.test.js @@ -0,0 +1,101 @@ +import assert from 'node:assert/strict' +import { test } from 'node:test' +import { createProjectServiceEvents } from '../src/host/project-service-events.js' +const seed = () => ({ projects: { p: { id: 'p', status: 'active', goal: 'Agreed goal' }, other: { id: 'other', status: 'active' } }, tasks: { task: { id: 'task', projectId: 'p', title: 'Current work', status: 'open' } } }) + +test('only acknowledged snapshots expose events; scoped cursors reconnect across restart', () => { + const bus = createProjectServiceEvents(), state = seed() + bus.load(state) + const start = bus.snapshot('p') + const next = structuredClone(state); next.tasks.task.status = 'working' + const pending = bus.prepare(next) + assert.equal(bus.query('p', { after: start.cursor }).events.length, 0) + bus.saved(pending) + const page = bus.query('p', { after: start.cursor }) + assert.equal(page.events.length, 1); assert.equal(page.events[0].data.status, 'working') + assert.equal(bus.query('p', { after: page.next }).events.length, 0) + assert.equal(bus.query('other').events.length, 0) + assert.throws(() => bus.query('other', { after: page.next }), /scope/) + const restarted = createProjectServiceEvents(); restarted.load(pending.snapshot) + assert.deepEqual(restarted.query('p', { after: start.cursor }), page) +}) + +test('uncommitted changes and private records cannot leak or perturb public Project event sequence', () => { + const bus = createProjectServiceEvents(), state = seed(); bus.load(state) + const initial = bus.snapshot('p') + state.projectAgents = { private: { notes: ['private-secret'], ownerUserId: 'owner', projectId: 'p' } } + state.projectConsultations = { private: { text: 'private-secret', projectId: 'p' } } + state.authSessions = { sensitive: { token: 'secret-token' } } + const privateSave = bus.prepare(structuredClone(state)); bus.saved(privateSave) + assert.equal(bus.snapshot('p').cursor, initial.cursor) + assert.equal(JSON.stringify(bus.query('p')).includes('private-secret'), false) + const candidate = structuredClone(state); candidate.tasks.task.status = 'accepted' + bus.prepare(candidate) // failed storage write; never acknowledged + assert.equal(bus.snapshot('p').tasks[0].status, 'open') + const recovered = bus.prepare(structuredClone(state)); bus.saved(recovered) + assert.equal(bus.query('p').events.length, 0) +}) + +test('bounded retention expires old cursors and snapshots require a consistent revision for later pages', () => { + let clock = 1000 + const bus = createProjectServiceEvents({ maxPerProject: 2, maxAgeMs: 1000, now: () => clock }), state = seed(); bus.load(state) + const old = bus.snapshot('p').cursor + for (let n = 0; n < 3; n++) { const next = structuredClone(state); next.tasks.task.status = 'step-' + n; bus.saved(bus.prepare(next)) } + assert.equal(bus.query('p').events.length, 2) + assert.throws(() => bus.query('p', { after: old }), error => error.code === 'bot/cursor-expired' && error.nextAction === 'project.snapshot') + assert.throws(() => bus.snapshot('p', { expectedRevision: 0 }), /changed/) + clock += 2000 + bus.saved(bus.prepare(structuredClone(state))) + assert.equal(bus.query('p').events.length, 1, 'only the new changed state survives age retention') + for (const options of [{ limit: 0 }, { limit: 101 }, { after: 'malformed' }]) assert.throws(() => bus.query('p', options)) +}) + +test('clock rollback cannot create a retention hole; malformed persisted streams fail closed', () => { + let clock = 5000 + const bus = createProjectServiceEvents({ maxPerProject: 3, maxAgeMs: 1000, now: () => clock }), state = seed(); bus.load(state) + let prepared + for (const [at, status] of [[5000, 'one'], [1000, 'two'], [5100, 'three']]) { clock = at; state.tasks.task.status = status; prepared = bus.prepare(structuredClone(state)); bus.saved(prepared) } + assert.deepEqual(bus.query('p').events.map(row => row.at), [5000, 5000, 5100]) + const restarted = createProjectServiceEvents(); restarted.load(prepared.snapshot) + const broken = structuredClone(prepared.snapshot), rows = Object.values(broken.projectServiceEvents) + delete broken.projectServiceEvents[Object.keys(broken.projectServiceEvents)[0]] + assert.throws(() => restarted.load(broken), /gap|duplicate/) + const duplicate = structuredClone(prepared.snapshot) + duplicate.projectServiceEvents.duplicate = rows[0] + assert.throws(() => restarted.load(duplicate), /gap|duplicate/) +}) + +test('committed Project facts include current role revision and native activation/exit facts without host data', () => { + const bus = createProjectServiceEvents(), state = seed() + state.projectAgentDefinitions = { p: { revision: 1, roles: [] } } + state.projectAttempts = { a: { id: 'a', projectId: 'p', taskId: 'task' } } + bus.load(state) + assert.equal(bus.snapshot('p').project.roleDefinitionRevision, 1) + const cursor = bus.snapshot('p').cursor, next = structuredClone(state) + next.projectAgentDefinitions.p.revision = 2 + next.projectNativeWorkers = { native: { id: 'native', projectId: 'p', taskId: 'task', attemptId: 'a', version: 'project-native-worker/v2', role: 'developer', status: 'bound', generation: 2, sessionKey: 'session', sessionId: 'physical', activationEffect: { id: 'activate' }, processScope: { unit: 'HIDDEN_HOST_DETAIL' } } } + const pending = bus.prepare(next) + assert.equal(bus.query('p', { after: cursor }).events.length, 0) + bus.saved(pending) + const events = bus.query('p', { after: cursor }).events + assert.equal(events.find(row => row.entityType === 'project').data.roleDefinitionRevision, 2) + assert.equal(events.find(row => row.entityType === 'execution').data.harness, 'flybot') + assert.equal(events.find(row => row.entityType === 'execution').data.activationEffectId, 'activate') + assert.equal(JSON.stringify(events).includes('HIDDEN_HOST_DETAIL'), false) +}) + +test('committed role approval proposals are public scoped facts without personal proposal contents', () => { + const bus = createProjectServiceEvents(), state = seed(); bus.load(state) + const cursor = bus.snapshot('p').cursor, next = structuredClone(state) + next.projectRoleProposals = { proposal: { id: 'proposal', projectId: 'p', action: 'add', role: 'analyst', status: 'pending', baseRevision: 1, definition: { instructions: 'HIDDEN_DEFINITION' } } } + next.projectAgentProposals = { personal: { id: 'personal', projectId: 'p', text: 'HIDDEN_PERSONAL_PROPOSAL' } } + const pending = bus.prepare(next); bus.saved(pending) + const events = bus.query('p', { after: cursor }).events + assert.equal(events.length, 1) + assert.equal(events[0].entityType, 'role-proposal') + assert.equal(events[0].data.role, 'analyst') + assert.equal(JSON.stringify(events).includes('HIDDEN_'), false) + const snapshot = bus.snapshot('p', { entityType: 'role-proposal' }) + assert.deepEqual(snapshot.entities.map(row => row.id), ['proposal']) + assert.throws(() => bus.snapshot('p', { entityType: 'private-agent' }), /Unknown public/) +}) diff --git a/patches/ds-bot/files/test/project-service-sse.test.js b/patches/ds-bot/files/test/project-service-sse.test.js new file mode 100644 index 00000000..55cf2d8c --- /dev/null +++ b/patches/ds-bot/files/test/project-service-sse.test.js @@ -0,0 +1,93 @@ +// Actual HTTP/SSE transport with explicit in-memory event/authorization fixtures. +import assert from 'node:assert/strict' +import { createServer } from 'node:http' +import { once } from 'node:events' +import { test } from 'node:test' +import { createWorkbenchGateway } from '../src/host/workbench-gateway.js' +import { createProjectServiceEvents } from '../src/host/project-service-events.js' +async function fixture(t, maxStreams = 4) { + const state = { projects: { p: { id: 'p', status: 'active' } }, tasks: { task: { id: 'task', projectId: 'p', status: 'open' } } }, seen = [] + let bus = createProjectServiceEvents({ maxPerProject: 2 }), allowed = true + bus.load(state) + const upstream = createServer(async (req, res) => { + let text = ''; for await (const chunk of req) text += chunk + const body = JSON.parse(text), input = body.payload + seen.push({ token: req.headers['x-flybot-member-session'], cookie: req.headers.cookie, name: input.name }) + let result + try { + if (!allowed || req.headers['x-flybot-member-session'] !== 'fixture-member') throw Object.assign(new Error('Fixture revoked'), { code: 'bot/forbidden' }) + if (body.endpoint !== 'project-development' || input.kind !== 'query' || input.projectId !== 'p') throw Error('Wrong service query') + result = { ok: true, value: { result: input.name === 'events' ? bus.query(input.projectId, input.payload) : bus.snapshot(input.projectId, input.payload) } } + } catch (error) { result = { ok: false, error: { code: error.code ?? 'bot/error', details: error.details } } } + res.writeHead(200, { 'content-type': 'application/json', 'set-cookie': 'private-owner=do-not-forward' }); res.end(JSON.stringify(result)) + }) + upstream.listen(0, '127.0.0.1'); await once(upstream, 'listening') + const gateway = createWorkbenchGateway({ upstreamOrigin: `http://127.0.0.1:${upstream.address().port}`, upstreamCookie: 'owner=private-owner', eventPollMs: 10, eventMaxStreams: maxStreams }) + gateway.listen(0, '127.0.0.1'); await once(gateway, 'listening') + const base = `http://127.0.0.1:${gateway.address().port}` + t.after(async () => { gateway.closeAllConnections(); upstream.closeAllConnections(); await Promise.all([new Promise(done => gateway.close(done)), new Promise(done => upstream.close(done))]) }) + function update(status) { state.tasks.task.status = status; const prepared = bus.prepare(structuredClone(state)); bus.saved(prepared); return prepared.snapshot } + return { base, seen, gateway, state, bus: () => bus, update, revoke: () => { allowed = false }, restart(snapshot) { bus = createProjectServiceEvents({ maxPerProject: 2 }); bus.load(snapshot) }, + open: (headers = {}, signal) => fetch(`${base}/api/project-events?projectId=p`, { headers: { 'x-flybot-member-session': 'fixture-member', ...headers }, signal: signal ? AbortSignal.any([signal, AbortSignal.timeout(5000)]) : AbortSignal.timeout(5000) }) } +} +async function readUntil(reader, match, initial = '') { + let text = initial + while (!match(text)) { const next = await reader.read(); if (next.done) break; text += new TextDecoder().decode(next.value) } + return text +} + +test('SSE uses the original authenticated query service and reauthenticates before subsequent facts', async t => { + const f = await fixture(t), response = await f.open() + assert.equal(response.status, 200); assert.match(response.headers.get('content-type'), /text\/event-stream/) + assert.equal(response.headers.get('set-cookie'), null) + const reader = response.body.getReader(), first = await readUntil(reader, text => text.includes('event: snapshot')) + f.revoke(); f.update('after-revocation') + const all = await readUntil(reader, () => false, first) + assert.ok(all.includes('event: access-revoked')) + assert.equal(all.includes('after-revocation'), false) + assert.equal(all.includes('private-owner'), false) + assert.ok(f.seen.every(row => row.token === 'fixture-member' && row.cookie === 'owner=private-owner')) + assert.deepEqual(f.seen.slice(0, 2).map(row => row.name), ['project.snapshot', 'events']) +}) + +test('SSE Last-Event-ID resumes ordered committed facts after restart without replaying the acknowledged event', async t => { + const f = await fixture(t), start = f.bus().snapshot('p').cursor + f.update('working'); const snapshot = f.update('blocked') + f.restart(snapshot) + const control = new AbortController(), response = await f.open({ 'last-event-id': start }, control.signal), reader = response.body.getReader() + const first = await readUntil(reader, text => (text.match(/event: fact/g) ?? []).length >= 2) + const facts = [...first.matchAll(/data: (.+)\n/g)].map(match => JSON.parse(match[1])).filter(row => row.type) + assert.deepEqual(facts.map(row => row.data.status), ['working', 'blocked']) + control.abort() + const secondControl = new AbortController(), resumed = await f.open({ 'last-event-id': facts[0].cursor }, secondControl.signal) + const second = await readUntil(resumed.body.getReader(), text => text.includes('event: fact')) + assert.equal(second.includes('"status":"working"'), false) + assert.ok(second.includes('"status":"blocked"')) + secondControl.abort() +}) + +test('expired SSE cursors require snapshots; credentials in URLs, missing headers and cross-scope IDs are refused', async t => { + const f = await fixture(t), old = f.bus().snapshot('p').cursor + for (const status of ['one', 'two', 'three']) f.update(status) + const expired = await f.open({ 'last-event-id': old }) + assert.equal(expired.status, 409) + assert.equal((await expired.json()).error.details.nextAction, 'project.snapshot') + const missing = await fetch(`${f.base}/api/project-events?projectId=p`) + assert.equal(missing.status, 401) + assert.equal((await fetch(`${f.base}/api/project-events?projectId=p&token=secret`)).status, 400) + const foreign = JSON.parse(Buffer.from(f.bus().snapshot('p').cursor, 'base64url').toString('utf8')) + foreign.p = 'another-project' + const cross = await f.open({ 'last-event-id': Buffer.from(JSON.stringify(foreign)).toString('base64url') }) + assert.equal(cross.status, 409) + assert.equal((await cross.json()).error.code, 'bot/conflict') +}) + +test('SSE connections are bounded and server shutdown aborts open streams', async t => { + const f = await fixture(t, 1), response = await f.open(), reader = response.body.getReader() + await readUntil(reader, text => text.includes('event: snapshot')) + const over = await f.open() + assert.equal(over.status, 429); assert.equal((await over.json()).error.code, 'bot/busy') + const closed = once(f.gateway, 'close'); f.gateway.close() + await readUntil(reader, () => false) + await closed +}) diff --git a/patches/ds-bot/files/test/project-sources.test.js b/patches/ds-bot/files/test/project-sources.test.js index 92265eab..61df34c6 100644 --- a/patches/ds-bot/files/test/project-sources.test.js +++ b/patches/ds-bot/files/test/project-sources.test.js @@ -19,7 +19,11 @@ function fixture(state) { rt.projectEffects = createProjectEffects(rt); inputs(rt); sources(rt); rt.projectDevelopment = createDevelopmentService(rt) return { rt, project, schedules, disk: () => disk, reads: () => reads, fail(value) { failure = value }, revoke() { revoked = true }, feed(value) { feed = value } } } -const command = (f, name, payload, actor = owner, requestId = 'request') => f.rt.projectDevelopment.endpoint('project-development', { version: 'project-development/v1', kind: 'command', name, projectId: 'p', requestId, payload }, actor) +// Domain/fault tests use the original aliases on the same facade so receipt-intent +// saves cannot consume the injected source-transaction failure. Real v1 source and +// response-loss recovery are covered in project-effects-runtime.test.js. +const sourceEndpoints = { 'source.subscribe': 'project-source-subscribe', 'source.poll': 'project-source-poll', 'source.disable': 'project-source-disable' } +const command = async (f, name, payload, actor = owner, requestId = 'request') => ({ result: await f.rt.projectDevelopment.endpoint(sourceEndpoints[name], { ...payload, projectId: 'p', requestId }, actor) }) const subscribe = async (f, payload = {}) => (await command(f, 'source.subscribe', { taskId: 't', threadKind: 'pull', number: 7, includeHistory: true, ...payload })).result.subscription const poll = (f, sub, actor = owner) => command(f, 'source.poll', { threadId: sub.threadId }, actor) diff --git a/patches/ds-bot/files/test/project-state-runtime.test.js b/patches/ds-bot/files/test/project-state-runtime.test.js index 27bde6ef..ba0e3545 100644 --- a/patches/ds-bot/files/test/project-state-runtime.test.js +++ b/patches/ds-bot/files/test/project-state-runtime.test.js @@ -37,9 +37,28 @@ test('original Project APIs persist Task/intake intent in SQL without model call assert.equal(disk.projects.p.goal, 'Retain the agreed product goal') assert.equal(disk.projects.p.scope, 'Original Project scope') assert.equal(modelCalls, 0) + const snapshotBody = { version: 'project-development/v1', kind: 'query', name: 'project.snapshot', projectId: 'p', payload: {} } + const baseline = (await first.rt.projectEndpoint('project-development', snapshotBody, actor)).result + const proposalBody = { version: 'project-development/v1', kind: 'command', name: 'task.propose', projectId: 'p', requestId: 'propose-through-service', + payload: { botId: 'chief', title: 'A second explicit Project change', criteria: 'Independent regression passes', impact: 'Improve the Project service' } } + const proposal = await first.rt.projectEndpoint('project-development', proposalBody, actor) + assert.equal(proposal.receipt.status, 'completed'); assert.equal(proposal.receipt.committed, true) + const proposedId = proposal.result.id + assert.equal(first.rt.state.tasks[proposedId].commandSource.commandReceiptId, proposal.receipt.id) + assert.equal((await first.rt.projectEndpoint('project-development', proposalBody, actor)).result.id, proposedId) + assert.equal(Object.keys(first.rt.state.tasks).length, 2) + const eventBody = { version: 'project-development/v1', kind: 'query', name: 'events', projectId: 'p', payload: { after: baseline.cursor } } + const facts = (await first.rt.projectEndpoint('project-development', eventBody, actor)).result + assert.deepEqual(facts.events.map(row => row.entityId), [proposedId]) + const receiptBody = { version: 'project-development/v1', kind: 'query', name: 'command.receipt', projectId: 'p', payload: { commandRequestId: proposalBody.requestId } } + assert.equal((await first.rt.projectEndpoint('project-development', receiptBody, actor)).result.id, proposal.receipt.id) await first.ctx.dispose() const second = await start() assert.equal(second.rt.stateStorage.status().backend, 'sqlite') + assert.equal((await second.rt.projectEndpoint('project-development', proposalBody, actor)).result.id, proposedId) + assert.equal(Object.keys(second.rt.state.tasks).length, 2) + assert.deepEqual((await second.rt.projectEndpoint('project-development', eventBody, actor)).result.events.map(row => row.entityId), [proposedId]) + assert.equal((await second.rt.projectEndpoint('project-development', body, actor)).result.tasks[0].taskId, task.id) assert.equal(second.rt.state.tasks[task.id].projectTask.admission.status, 'proposed') assert.equal(modelCalls, 0)