Repository navigation
[Rewrite] Umbrella: nostream 4.0 — Rust + LMDB, factory-managed #860
Description
Purpose
Umbrella issue for the complete rewrite of nostream from TypeScript/Node.js/PostgreSQL to Rust with an LMDB-based storage backend (strfry-style). Led by a factory manager agent from 0 to 100%.
Ground rules (frozen)
- Capture point:
242a3001d77149e4d70151ec03060c4ef44a1547(nostream@3.4.0, 2026-10-13T19:31:52Z). The plan absorbs every change onmainthrough 2026-10-15T00:00:00Z — measured: 24 commits between capture and tip, including nostream@3.5.0/3.5.1 (NIP-77 negentropy wire protocol, settings URL validation, security bumps). Protocol currency is judged against the NIP registry as of the capture date, then re-checked at cutover. - Out of scope: the website (
website/, the Astro marketing site) — it stays TypeScript. - Protocol compliance: the rewrite must support every NIP the relay advertises at capture (
supportedNips: 1,2,3,4,9,11,12,13,14,15,16,17,20,22,25,28,33,40,42,43,44,45,50,65,77 — 25 NIPs) plus any NIP change merged through the capture window; Phase 0 diffs the registry against this list and adds issues for the gaps. - Costs: every issue carries an Expected Cost (USD) on the project board; on completion the Actual Cost (USD) is recorded from the factory's own spans (measured baseline: job 001 — a ~165-line diff across 6 files — cost 2.2 h factory wall time).
- Hardware: the factory runs on chappie (this host: orchestration, reviews, LMDB benchmark rig); tauceti (Ada's host) takes heavy Rust release builds and long integration suites via a shared worktree per phase. The split is recorded per issue in the project's Host field.
- Oracle: the existing test suite (33,053 lines of TS tests) is the porting oracle — a Rust module is done when it reproduces the TypeScript module's observable behaviour, verified by the parity tests in Phase 6 and, where they exist, the original spec tests re-expressed in Rust.
Factory manager's mandate
- Owns the board: moves issues through Todo → Costed → In progress → In review → Done/Blocked, sets Expected/Actual Cost and Host fields.
- Monitors spend per phase against the plan; when a phase exceeds its budget by >25% or a milestone slips, adjusts the project plan, milestones and timelines on the board and records the adjustment in this issue's comments.
- Runs each issue through the factory loop (recon → spec → build → review → approve) with the six lenses; escalates per the factory's needs-human rules.
Project board
https://github.com/users/cameri/projects/4
Issue tree
See the phase issues #860–#868 (created below as sub-issues of this one).
Labels: rewrite on everything; rust, lmdb, mcp, factory by area.
Activity
chappie-daemon commentedon Oct 17, 2026
Phase 0 (job
001-nip-audit, #861) — delivered, awaiting review.rounds-exhaustedafter the review was halted at round 5. Eight round-4 findings were fixed in the escalated head and no lens reviewed that head;decision.mdin the job folder records what was and was not reviewed.One planning input from this job. The rewrite plan's premise that jobs branch from
rewrite/maindoes not hold: norewrite/mainexists in this repository (47 remote heads checked), so this job branched frommain's tip811fbb5. The PR body carries the integration note — the audit file must be carried across, or the cut re-pointed, at the moment Phase 1 code jobs start. Worth settling before the first code job, not after.Cost data suggests the per-issue estimates are conservative by roughly an order of magnitude for documentation-shaped work; no change made to the board's expected costs, since one sample is not a basis for rescaling the plan.
chappie-daemon commentedon Oct 17, 2026
Cost calibration — first measured job.
#861's Phase 0 job landed at 0.046× its planning anchor ($3.71 measured against $80 expected). The anchor was set before any job had run under this process, so it was a placeholder, not an estimate.
Method, from here on: every phase's expected cost is recalibrated from that phase's first measured job, not from a global fudge factor applied to the original anchors —
with scope stated in the units the measurement is per (sessions and requests, split by stage), and the anchor replaced rather than scaled. Phase 0's calibration input: 30 sessions, 1,362 requests, $0.124/session across recon/spec/build/review-scoping/lenses, of which lenses are 60.2%.
No board change made yet: one job calibrates Phase 0 only, and Phase 1 has no measured job. When Phase 1's first job closes, its anchor is replaced by the same formula and the replacement is recorded here. This is now standing cost duty — every completed issue is reported actual vs expected, and an anchor is only allowed to stand while its phase has no measurement.
chappie-daemon commentedon Oct 19, 2026
Phase 1 anchor replaced — first measured job.
#862 (job
005-rust-foundation) closed at actual $14.7016 against a $1,200 anchor — 1.2% of the anchor — over 15 sessions and 1,437 requests (recon 1, spec 2, build 1, review and confirmations 11). Stage split and thefactory-spansinvocation:projects/nostream/.factory/jobs/005-rust-foundation/cost.md.The rule recorded above applies unchanged:
Phase 1 input: 15 sessions, $14.7016 → $0.9801/session.
Project 4 changes, applied 2026-10-19:
nostream-coreSo, plainly: the $1,200 Phase 1 anchor is retired. The $900 and $200 figures were placeholders of the same family — neither was ever an estimate — and neither survives on the board. #862 keeps $1,200 as its expected only because that is the anchor it was actually run against.
Caveat recorded with the replacement. #863 carries more acceptance surface than #862 did: a 1,000-event differential fixture and 328 TypeScript unit tests re-expressed against an 842-line crate carrying 23 stubs. $14.70 is the calibrated floor, not a forecast with headroom. A budget overrun past 25% escalates per the operating rule, and the same formula re-applies from #863's measurement when it closes.
Still unreplaced: #864 ($2,400), #865 ($2,000), #866 ($1,600), #870 ($450), #867 ($1,400), #868 ($1,200), #869 ($600) — each is replaced when its phase's first job closes, not before.
Two housekeeping notes: the Phase 0 item (#861) still shows Expected $80 against Actual $3.71, which is the same placeholder situation — the earlier note deferred it and nothing has replaced it yet; and this umbrella's own Project item also carries Expected Cost USD 1,200. I have not touched either: the umbrella is not a phase, so its field should be cleared or set to the sum of its children, and that is Cameri's call.
chappie-daemon commentedon Oct 19, 2026
Phase 1, second measured job — #863 closed at
b324ef5e(PR #907), and the cost figure is not measurable on this host.The pricing gap is the finding: the run was served by
opencode-deepseek-v4.1-flash, which has no price configured, so everyusagerecord in all 19 of the job's transcripts carriescost: 0.factory-spansappended its line honestly ($0.00) — that is an absence of price, not a free job, and the Project item's Actual Cost USD is deliberately left empty rather than filled with a guess.What was measured: 19 sessions, 1,572 requests, 2.11M input / 1.43M output tokens, 250.3M cache-read, wall ~1h50m of agent time (recon 9m, spec 10m, build 19m46s + three resumes, four review rounds ≈20/12/8/7m, approve 5m30s).
Proxy for the calibration, labelled as one: at #862's blended $0.0102/request, 1,572 requests ≈ $16 — inside #863's $55 forecast and its $40–70 band. The formula cannot be re-applied from this job until the served model carries a price; when it does, this comment is the input to re-derive
expected(Phase 1)from two jobs instead of one.Two harness defects recorded with it: missing pricing needs an explicit "unpriced" marker so $0.00 is never read as a result, and
factory-spans' lower bound anchored on the job's firstplanline — which excluded this job's own recon, spec and first builder run (three sessions, ~500 requests).chappie-daemon commentedon Oct 19, 2026
Program-order adjustment: #864 (Phase 3, storage seam + LMDB) runs before #865 (Phase 2, relay handlers), on measured evidence.
The milestones put #865 in Phase 2, so the board order opened it first. Recon for #865 (
007-relay-handlers, stopped before the spec, no branch left behind) measured why that order cannot hold:EventRepositorydeclares 4 of 11 methods andUserRepository3 of 7 —upsert,find_by_filters,delete_by_pubkey_except_kinds,is_vanished/set_vanished/revoke_admissionandreport.create_manyare missing — so even a test double cannot serve REQ/EVENT. [Rewrite P3] Storage seam + LMDB backend (strfry-style) + PG→LMDB migration #864 is a hard prerequisite for [Rewrite P2] nostream-relay: wire handlers, strategies, rate limits #865's acceptance as written, and soft only for a handler-only implementation.DATABASE_URLunset — and the Cucumber suite'sBeforeAllneeds the same database.So: #864 opens next (its own anchor $2,400 stands only because Phase 3 has no measured job; it is replaced, not scaled, from its measurement). #865 is not cancelled — it resumes after the storage seam exists, with the frame fixture and the 774-case oracle already identified. This is a disposition change, recorded here before the job opens, per the charter.
chappie-daemon commentedon Oct 19, 2026
Phase 3 scope split (#864), recorded before the build — expected cost replaced, not absorbed.
The issue bundles five surfaces under a $2,400 anchor set when no Phase-3 job had run. Recon measured what each surface is (traits 51 methods across nine interfaces; 37 knex migrations; LMDB store 2.5–4k LOC; NIP-50 300–600; migration tool 600–1,000; benchmark rig with no corpus existing), and the spec delivered the first slice:
migrate pg-to-lmdb(follow-up)The $2,400 was replaced on the Project item with $1,170 for this job, and the four remainders are named, sized and preconditioned in
spec.md§6. They will be carried as their own items with their own expected costs rather than folded back into this one. Phase 3 still has no measured job — this job's close supplies the first, and the same formula then replaces these figures.}