Persist Project state and Effects with guarded SQLite migration

LESSONS.md

@@ -665,6 +665,10 @@ Seen 1× while implementing #2179. Reusing old group aggregation would accept ca
 
 #2179 integration regression: `git rev-parse MERGE_HEAD` returns only the first parent of an octopus merge. Validate every bounded registered MERGE_HEAD entry before sealing; an admitted first source cannot conceal an unapproved second parent. Preserve the staged index on rejection.
 
+### 2026-11-03 — Migration must distinguish unfinished import from missing current data
+
+An interrupted prepared SQL migration may recover its verified import; an active database that disappears must fail closed, not silently revive the old import. Publish a newer-version JSON guard before initialization, validate one authority, and retain a fenced resumable rollback marker across partial moves. A transaction commit must precede Effects acknowledgement without a fallible post-commit read masking that proof.
+
 
 ### 2026-11-03 — Public Chief cannot borrow its approver's identity
 
@@ -747,3 +751,7 @@ Seen 1× (composed actual DSH native input failure). A canonical claim keeps the
 ### 2026-11-03 — Fence old-plan decisions as well as external effects
 
 Seen 1× (late Task requirement response regression). Stopping on the next scheduler pass alone leaves a window for an old turn to transfer/split. Bind current claims to the Task requirement revision and recheck after authorization; use actual old Attempt snapshot revisions for compatible prior claims. Keep publication evidence fences too.
+
+### 2026-11-03 — State maps do not encode chronology
+
+Seen 1× (actual SQLite native return probe). Two valid Attempts survived, but UUID-key SQL iteration put the successor first and an insertion-order assertion compared it to itself. Use persisted predecessor references for chronology; preserve explicit history positions. Storage order must not decide current execution or evidence.

PROJECT.md

@@ -52,60 +52,7 @@ Detailed S1–S6 implementation/recovery history is [archived verbatim](docs/his
 
 ## Historical upstream Project State
 
-## Mission
-
-FlyBot is a local team of long-lived AI Bots that runs on DeepSeek Harness (DSH), in your own
-browser, with your own model keys. One person talks to one Bot — the **Chief** — and the team takes
-the request apart and comes back with one answer. The Bots keep their own memory, working folder,
-model and schedules between turns, so work outlives a single chat.
-
-The project's mission is to keep that team reliable and inspectable: every Bot acts on evidence,
-stays inside its goal, and its result can be checked by a person afterwards. The product is the app
-in this repository plus the patches that pin it to upstream `ds-bot`; it is not a hosting service.
-
-`AGENTS.md` states the two halves the code is built from: pristine upstream `ds-bot/` (never edited),
-and `core/` — `ds-bot/` plus FlyBot's ordered patches — which is what ships and what the tests run.
-The shipped features are the DSH plugin in `core/` and the peer plugins `plugins/social`,
-`plugins/tree`, `plugins/timeline` and `plugins/workflow`.
-
-## Owner and maintainer roles
-
-Today the project has one owner and a small number of human maintainers; the Bots work under the
-owner's delegation. This is the state of the repository as of 2026-10-29, not the target model of
-[#1550](https://github.com/ThinkFlowLab/FlyBot/issues/1550).
-
-| Role | Who | Decides |
-| --- | --- | --- |
-| **Owner** | Danyang, GitHub [`Birfy`](https://github.com/Birfy) (repository admin) | Mission, governance policy, roles and membership, project Bots, compute and resources, budgets, autonomy boundaries, and every matter reserved to the owner in [docs/governance.md](docs/governance.md) |
-| **Maintainer** (human) | People with repository write access — today the owner and [`hsliuustc0106`](https://github.com/hsliuustc0106) | Major decisions under [docs/governance.md](docs/governance.md): roadmap and architecture changes, changes to the governing documents, multi-subsystem contracts. Records a verdict through the review policy, never through an admin merge |
-| **Chief** (AI) | The owner's Main Bot, writing as `Birfy` | Plans and routes work, splits a request into role tasks, **classifies** each decision as ordinary / major / severe with its rationale, convenes a discussion when one is needed, and escalates to the owner. Not the ultimate authority and not the owner |
-| **Routine Bots** (AI) | `Triage`, `Dev`, `Dev2`–`Dev4`, `Reviewer`, `Reviewer2`, `Release`, `SelfTest`, `Docs`, `Maintainer` — the names in [`writers`](.github/agent/review-policy.json) | Only their routine's job: see [Routines](AGENTS.md#routines). A Bot records a conclusion only where its routine owns that conclusion, and the implementer never writes the verdict on its own change |
-| **Contributors** (human) | Anyone who opens an issue or a pull request | Proposals, bug reports, code. No default approval right; a contribution merges only through the gates in [AGENTS.md](AGENTS.md) |
-
-Deliberately **not** claimed here, because it is not shipped: a Maintainer console, a Member role, a
-Project Assistant, per-person GitHub attribution for Bots, and the multi-human authority contract in
-[docs/subsystems/team.md](docs/subsystems/team.md) (planned; team mode stays disabled until its
-authorization gates pass). Every Bot still writes as the one shared `Birfy` identity, so a
-`Written-by:` line is a claim the gates can check for presence and name, not a signature.
-
-## Resources covered
-
-- **This repository** — `https://github.com/ThinkFlowLab/FlyBot`, the source of record. `main` is
-  the live branch; pull requests merge into it only through the merge queue.
-- **Code** — `core/` (shipped plugin, source of truth), `patches/ds-bot/` and `core/client.js`
-  (generated from `core/`), `plugins/*` (peer plugins), `scripts/` and `core/scripts/`.
-- **Rules and knowledge** — `AGENTS.md` (operating contract), `CONTRIBUTING.md`, `LESSONS.md`
-  (observed failures and promotion to standards), `docs/` (architecture, standards, runbooks),
-  `decisions/` (dated decisions), `PROJECT.md` and `ROADMAP.md` (this page and the plan).
-- **Automation** — `.github/workflows/` and `.github/scripts/` (CI, risk tier, review gate, merge
-  gate, enqueue guard), `.github/agent/` (risk rules and the review policy).
-- **Authoritative runtime state elsewhere** — GitHub issues, pull requests and CI runs are the
-  authority for work state; task boards, approvals and per-Bot memory live in FlyBot's own storage,
-  never in this repository. `docs/` pages are informative only where a source above decides
-  something.
-- **Not in this repository** — the upstream `ds-bot` project (pinned in `ds-bot.pin.json`, read-only
-  here) and the self-hosted Bot Social server used by the social plugin. The fork
-  `Birfy/FlyBot` is an archive: nothing merges into it.
+Historical upstream mission, named roles and resource inventory are [preserved verbatim](docs/history/2026-11-03-upstream-project-inventory.md). Current fork ownership and capabilities follow the active state and dated decisions above. The governing and automation boundaries below remain active.
 
 ## Governing boundaries
 
@@ -372,4 +319,10 @@ New shared Tasks have Chief admission and versioned requirements. Existing Task/
 
 #2179 follow-up: Chief dependency waivers/substitutions retain exact outcome versions and explicit judgement evidence. Accepted child code enters the existing durable integration gates before parent execution; source admission is not merging or parent acceptance. Actual Git multi-parent sealing tests use deterministic child acceptance fixtures. Full real-provider parallel collaboration and native physical restoration remain pending.
 
+### 2026-11-03 — Transactional state (#2182)
+
+Opt-in SQL retains the original Store/services and saves domain facts plus Effects atomically. JSON becomes a newer-version migration guard, with verified import/recovery and offline export/rollback; no dual writable snapshots. Histories are normalized and large immutable payloads hash-referenced. Routine state updates/views make no additional model calls or Project-position changes. Full-state hydration/scanning remains; storage growth is distinct from #2177 context budgets. No live migration or provider-loop claim.
+
 Composition checkpoint: current role governance/native exit, bounded context and canonical Task waits now share one execution chain. Native input fences logical role/physical worker separately; Task requirement revision changes stop the old Attempt through existing pool release. Actual composed/native acceptance and remaining API/storage compatibility are still under verification, not deployment.
+
+SQLite composition now has an actual DSH native return probe: canonical Chief intake, persisted activation/exit, original role context and single-resource handoff are read from committed SQL snapshots. Inference remains deterministic; no production migration or full-provider workflow acceptance is claimed.

ROADMAP.md

@@ -448,6 +448,8 @@ New shared Tasks have Chief admission and versioned requirements. Existing Task/
 
 #2179 follow-up: Chief dependency waivers/substitutions retain exact outcome versions and explicit judgement evidence. Accepted child code enters the existing durable integration gates before parent execution; source admission is not merging or parent acceptance. Actual Git multi-parent sealing tests use deterministic child acceptance fixtures. Full real-provider parallel collaboration and native physical restoration remain pending.
 
+#2182 implements opt-in transactional SQLite through the existing Store, exact-identity migration, process-death commit tests and recoverable offline rollback. Retain configured Chief semantic decisions while routine facts/queries remain deterministic. Public scoped event/receipt service #2185, legacy entrypoint cutover #2186 and final real deployment/README #2189 remain subsequent stages; current production state is unchanged.
+
 Native prior-host reconciliation now reuses verified supervisor process-group facts; native v2 registered-work/lineage and shared adapter-exit regressions guard against premature initiative transfer. In-flight current-rights abort and composed-stack acceptance remain active.
 
 Canonical ongoing role/resource authority checks now reuse native step/SDK input and the existing scheduler + pool stop. Frozen role context remains stable; current revoked rights stop execution, with unknown exit held. Compose descendants and verify parent Task resume/API projections next.

changelog.d/2182.added.md

@@ -0,0 +1 @@
+Add opt-in transactional SQLite state with atomic Effects, normalized history, hash-verified payloads, guarded JSON migration and recoverable offline export/rollback; retain existing Project APIs and deterministic state updates. Verify canonical Chief/native Task-role continuation, activation and exit proofs against committed SQLite snapshots without relying on map insertion order.

core/scripts/state-storage.mjs

@@ -0,0 +1,25 @@
+// Offline private operator interface; no account credentials are accepted here.
+import { resolve } from 'node:path'
+import { readFileSync } from 'node:fs'
+import { exportSqliteState, rollbackSqliteState } from '../src/host/project-state-migration.js'
+
+try {
+  const [command, ...args] = process.argv.slice(2), options = {}
+  if (!['inspect', 'export', 'rollback'].includes(command)) throw new Error('Usage: node core/scripts/state-storage.mjs inspect|export|rollback --home PATH [--migration ID] [--output PATH] [--resume]')
+  for (let i = 0; i < args.length; i++) {
+    const key = args[i]
+    if (!['--home', '--migration', '--output', '--resume'].includes(key) || Object.hasOwn(options, key)) throw new Error('Unknown or duplicate migration option')
+    if (key === '--resume') options[key] = true
+    else if (!args[i + 1] || args[i + 1].startsWith('--')) throw new Error('Migration option requires a value')
+    else options[key] = args[++i]
+  }
+  if (!options['--home']) throw new Error('An explicit private --home is required')
+  const home = resolve(options['--home'])
+  let result
+  if (command === 'inspect') {
+    const guard = JSON.parse(readFileSync(resolve(home, 'state.json'), 'utf8'))
+    result = { backend: guard.storage === 'sqlite' ? 'sqlite' : 'json', phase: guard.phase ?? null, migrationId: guard.migration?.id ?? null }
+  } else if (command === 'export') result = exportSqliteState({ home, expectedMigrationId: options['--migration'], output: options['--output'] ? resolve(options['--output']) : null })
+  else result = rollbackSqliteState({ home, expectedMigrationId: options['--migration'], resume: options['--resume'] === true })
+  process.stdout.write(JSON.stringify(result) + '\n')
+} catch (error) { process.stderr.write(`State storage operation refused: ${error.message}\n`); process.exitCode = 1 }

core/src/host/project-state-migration.js

@@ -0,0 +1,84 @@
+// Offline operator tooling. No endpoint grants migration privileges to a Bot.
+import { createHash, randomUUID } from 'node:crypto'
+import { existsSync, readFileSync, renameSync } from 'node:fs'
+import { join } from 'node:path'
+import { homeLock } from './home-lock.js'
+import { openSqliteState } from './project-state-sqlite.js'
+import { writeDurableStateFile } from './project-state-storage.js'
+
+const digest = text => createHash('sha256').update(text).digest('hex')
+const fail = message => Object.assign(new Error(message), { code: 'bot/state-migration' })
+function exclusively(home, action) {
+  const path = join(home, 'state.lock')
+  if (existsSync(path) && JSON.parse(readFileSync(path, 'utf8'))?.pid === process.pid) throw fail('Stop the current FlyBot host before migration')
+  const lock = homeLock(path)
+  if (lock.claim() !== undefined) throw fail('Stop the active FlyBot host before offline migration')
+  try { return action() } finally { lock.release() }
+}
+function guardOf(home, expectedMigrationId) {
+  const guard = JSON.parse(readFileSync(join(home, 'state.json'), 'utf8'))
+  if (guard.version !== 3 || guard.storage !== 'sqlite' || guard.phase !== 'active' || !expectedMigrationId || guard.migration?.id !== expectedMigrationId) throw fail('The active SQLite migration ID changed; inspect before exporting or rolling back')
+  return guard
+}
+function snapshot(home, expectedMigrationId) {
+  const guard = guardOf(home, expectedMigrationId)
+  const db = openSqliteState({ path: join(home, 'state.sqlite'), readOnly: true })
+  try {
+    const metadata = db.metadata()
+    if (metadata.migration?.id !== guard.migration.id || metadata.migration?.sha256 !== guard.migration.sha256) throw fail('SQLite and guard identities differ')
+    const state = db.read()
+    if (state.version !== 2) throw fail('Unsupported exported state version')
+    guardOf(home, expectedMigrationId)
+    return { text: JSON.stringify(state, null, 2), revision: metadata.revision }
+  } finally { db.close() }
+}
+export function exportSqliteState({ home, output, expectedMigrationId }) {
+  if (!output) throw fail('Choose a private export output file')
+  return exclusively(home, () => {
+    const current = snapshot(home, expectedMigrationId)
+    writeDurableStateFile(output, current.text, { exclusive: true })
+    return { backend: 'sqlite', revision: current.revision, sha256: digest(current.text), output }
+  })
+}
+export function rollbackSqliteState({ home, expectedMigrationId, resume = false }) {
+  return exclusively(home, () => {
+    const markerPath = join(home, 'state-storage-rollback.json'), path = join(home, 'state.sqlite'), statePath = join(home, 'state.json')
+    const previous = existsSync(markerPath) ? JSON.parse(readFileSync(markerPath, 'utf8')) : null
+    let marker
+    if (previous?.phase === 'prepared') {
+      if (!resume || previous.migrationId !== expectedMigrationId) throw fail('A prepared rollback exists; explicitly resume its exact migration ID')
+      marker = previous
+    } else {
+      if (resume && previous?.phase === 'completed' && previous.migrationId === expectedMigrationId) return { ...previous, already: true }
+      if (resume) throw fail('No matching prepared rollback exists')
+      const current = snapshot(home, expectedMigrationId), id = randomUUID()
+      marker = { version: 'flybot-storage-rollback/v1', phase: 'prepared', id, migrationId: expectedMigrationId, revision: current.revision,
+        exportFile: `state.rollback-${id}.json`, archiveFile: `state.sqlite.rollback-${id}`, sha256: digest(current.text), preparedAt: Date.now() }
+      writeDurableStateFile(join(home, marker.exportFile), current.text, { exclusive: true })
+      writeDurableStateFile(markerPath, JSON.stringify(marker, null, 2))
+    }
+    if (marker.version !== 'flybot-storage-rollback/v1' || !/^[a-f0-9-]{36}$/.test(marker.id) || marker.archiveFile !== `state.sqlite.rollback-${marker.id}` || marker.exportFile !== `state.rollback-${marker.id}.json` || !/^[a-f0-9]{64}$/.test(marker.sha256)) throw fail('Rollback recovery marker is malformed')
+    const text = readFileSync(join(home, marker.exportFile), 'utf8')
+    if (digest(text) !== marker.sha256 || JSON.parse(text).version !== 2) throw fail('Rollback export hash changed')
+    const currentText = readFileSync(statePath, 'utf8')
+    if (digest(currentText) !== marker.sha256) {
+      guardOf(home, expectedMigrationId)
+      if (existsSync(path)) {
+        const current = snapshot(home, expectedMigrationId)
+        if (current.revision !== marker.revision || digest(current.text) !== marker.sha256) throw fail('SQL state changed after rollback preparation')
+      } else if (!existsSync(join(home, marker.archiveFile))) throw fail('Rollback database archive is missing')
+      // Keep DB/WAL/SHM names paired. A crash between moves leaves the marker and
+      // verified export; resume finishes it before any runtime can write again.
+      for (const suffix of ['', '-wal', '-shm']) {
+        const source = path + suffix, target = join(home, marker.archiveFile) + suffix
+        if (!existsSync(source)) continue
+        if (existsSync(target)) throw fail('Rollback archive already contains a conflicting file')
+        renameSync(source, target)
+      }
+      writeDurableStateFile(statePath, text)
+    }
+    marker = { ...marker, phase: 'completed', completedAt: Date.now() }
+    writeDurableStateFile(markerPath, JSON.stringify(marker, null, 2))
+    return { ...marker, backend: 'json', removeStateStorageSqliteConfig: true }
+  })
+}