Repository navigation
[Rewrite P1] nostream-core: event model, crypto, filter matching #863
Description
Parent: #860
Milestone: Phase 1: Rust foundation
Host: chappie · Expected cost: $55 calibrated (the original $900 was an unreplaced Phase-1 anchor; replaced per the #860 rule from #862's measured $14.70, which is the floor. The 55 is the $40–70 forecast's midpoint.)
nostream-core: the event model and the NIP-01 semantics everything else sits on.
- Event type: id/pubkey/created_at/kind/tags/content/sig with serde (hex <-> bytes zero-copy where the wire format allows), 32-byte
EventId/Pubkey/Signaturenewtypes - Schnorr/secp256k1 signing + verification (
secp256k1crate,bitcoin_hashes/sha2), verifying the id hash construction (measured NIP-01 serialization) — property tests against known vectors - NIP-01 filter matching (
SubscriptionFilterport fromsrc/@types/filter-schema.ts): tags, kinds, authors, ids, since/until,#e/#p/#a/#d, search (NIP-50) — the TS matcher's truth table is the oracle (portfilter-schema.tstests 1:1) - Subscription filter normalisation + the negentropy-range helpers NIP-77 needs (port
src/utils/nip77.ts) - NIP-44 encryption, NIP-40 expiration, NIP-13 PoW difficulty calculation, NIP-22/45 COUNT semantics (data model parts)
- Reference-time handling and created_at monotonicity helpers the relay's replaceable-event logic needs
Acceptance: every TS unit test for filters/crypto/kinds re-expressed and green in Rust; differential test: 1,000 sampled events from the TS pipeline produce identical id/sig/match decisions in the Rust core.
Activity
chappie-daemon commentedon Oct 19, 2026
Factory job
006-nostream-coreopened: baserewrite/main@384b283, branchfactory/006-nostream-core, worktreeprojects/nostream-worktrees/006-nostream-core, factory 0.10.6, expected $900. Stage: recon. Project item moved to In progress.chappie-daemon commentedon Oct 19, 2026
Cost basis, stated before the spec stage.
The $900 in this issue and on its Project 4 item is an unreplaced pre-measurement anchor — not an estimate, and not a recalibrated figure. It was set with the rest of the plan's anchors, before any job had run under this process.
It was replaced on 2026-10-19 by the Phase 1 recalibration from #862 — 15 sessions × $0.9801/session = $14.70 (actual $14.7016 on that job). The replacement, its arithmetic and its caveat are recorded on the umbrella: #860
For the spec's adequacy finding: the calibrated figure is $14.70, and it is a floor rather than a forecast with headroom — this job's acceptance (a 1,000-event differential fixture plus 328 TypeScript unit tests re-expressed against an 842-line crate carrying 23 stubs) is a larger surface than the job that produced the measurement. Actual cost is filled on the Project item at close, from
factory-spans. Recon material for the spec, including the measured base line (157 workspace tests green, 4 in this crate), is in.factory/jobs/006-nostream-core/recon.md.chappie-daemon commentedon Oct 19, 2026
Spec stage complete (
spec.md, 1,124 lines): 14 fixed decisions, 23 requirements, 20 acceptance criteria (C1–C20), 16 files in scope, oracle = a 1,000-event differential fixture generated from the capture plus one Rust parity module per TS spec. Scope ruled out with reasons:validate_event(57 TS cases),adaptive_pow, NIP-77 frames, the twelve NIP stubs.Cost finding, plainly: the $900 on this issue is an unreplaced Phase-1 anchor, not a calibrated figure. #862 measured $14.70 for a smaller config-shaped crate, so this job's calibrated expectation is roughly $40–70 — the $900 must not be used to grow scope. The board's replacement ($14.70 for #863, recorded on #860 at 11:06Z) is the calibrated floor. Build round 1 now dispatched.
chappie-daemon commentedon Oct 19, 2026
Build round 1 complete: STATUS: READY, six commits on
factory/006-nostream-core(not pushed), 29 files, +6,199/−102.cargo test --workspace --all-features→ 529 passed / 0 failed (base 157); both clippy passes clean; release build green; 17 labelled stubs remain for the out-of-scope modules and zerotodo!/unimplemented!.Differential oracle landed:
gen-ts-events.mjs+ a committed 902 KBts-events.json(1,000 events + 50 negatives, 16 probe filters) +ts_parity.rs, replaying canonical bytes, id, signature, PoW and 16,800 filter decisions identically; regeneration is byte-identical. One divergence found and recorded rather than hidden: thenegentropycrate answers a frame-truncated case from the current range's upper bound where the capture trackswrittenUpTo— sessions still converge, one extra round; C11's three reference vectors pass. Review scoping now.}chappie-daemon commentedon Oct 19, 2026
Review round 1 closed: security PASS, performance PASS, code CHANGES, architecture CHANGES. Four lenses ran; ux/ui stood down (no rendered surface in the diff).
Findings, all concrete: (code) NIP-44
validate_payloadfails open — a base64 string Rust refuses is reported valid, where the capture answers invalid, and the 7 parity cases miss it; a fabricated zod message (<= 18446744073709551615) for kinds/limit the capture actually accepts; length bounds counted in chars where zod counts UTF-16 units. (architecture) theFilternarrowing broke settings acceptance — amirroringfilter withkinds: [[10000,19999]]loaded at base and in the capture, now falls back to defaults with an unnamed$finding; andlib.rscontradictstime.rsand ADR 0002 over who owns the created_at bounds.Security measured independently (5 probe crates in the builder container): no unwrap/panic before
#[cfg(test)], hex newtypes reject before decoding, NIP-44 MAC checked before decryption, noSettingsPubkey→Pubkeypath. Performance measured: 16,800matchescalls in one libtest tick (< 0.6 µs each), schnorr verify 35-43 µs, id ≤ 9.5 µs, release profile healthy. Builder round 2 dispatched with seven fixes.}chappie-daemon commentedon Oct 19, 2026
Builder round 2 complete: STATUS: READY, nine appended commits (
64682ff…78f6ec4), 14 files, +426/−35.cargo test --workspace --all-features→ 542 passed / 0 failed (round 1: 529), both clippy passes clean, release build green, tree clean.All seven round-1 findings fixed with a proving test each: NIP-44 no longer fails open (decoder configured to Node's leniency; 33 cases incl. the one the old tests missed); the u64 representability refusal now has its own honest message and an ADR entry instead of a fabricated zod string; length bounds count UTF-16 units (emoji cases added);
mirroringkind-range filters load again with three new config tests; core/relay/ADR now agree that core owns thecreated_atpredicate and relay the policy;sign_eventsets the author so a mismatched pubkey cannot be signed; NIP-40 reads theNumber()forms (0x10,0o10,1e3, whitespace). Four lenses now re-review the diff6eeb3b4..78f6ec4, scoped to their own findings.}chappie-daemon commentedon Oct 19, 2026
Review round 2 closed: security PASS, performance PASS, code CHANGES, architecture CHANGES. All five round-1 findings confirmed resolved by the lenses that raised them (NIP-44 fail-open proved closed by revert-testing and by a 262-payload corpus run against the capture's own validator; the u64 message, UTF-16 bounds, mirror-filter acceptance and created_at ownership likewise).
Round-2 residue, both real:
decryptstill uses the strict base64 engine and refuses a payload the capture decrypts whilevalidate_payloadaccepts it; andu64::MAX as f64is 2^64, so18446744073709551616saturates instead of being refused. Architecture's three are record accuracy: theinto_filterdoc presents the range drop as the wire crossing when the capture forwards ranges in the relayed REQ; the mirror-filter rationale misquotessubscription.ts:8; and ADR 0002 has an unfireable trigger plus two divergences without one. Two security residues are also cheap and queued:0x+10is accepted where Node gives NaN, and U+FEFF is not stripped where the capture accepts a BOM.Performance measured the UTF-16 fix where it lands (validate_filter, per subscription filter): 300–500 ns per 1024-unit ASCII string, ~1.5 µs at the schema bounds,
Filter::matchesbyte-identical to round 1. Builder round 3 dispatched (resumes 2 of 5).}chappie-daemon commentedon Oct 19, 2026
Builder round 3 complete: STATUS: READY, five appended commits (
84243e4…7429381), eight files, +182/−44. Workspace 545 passed / 0 failed, both clippy passes andfmtclean, release build green, tree clean, fixture digest unchanged.Six round-2 findings fixed with a proving test each:
decryptis now as lenient asvalidate_payload(34 NIP-44 tests, including the non-canonical re-encoding the capture decrypts, with the MAC still verified over decoded bytes); the representability boundary is exclusive at 2^64 via a shared constant (2^64 refused, 2^64−2048 accepted);into_matcher_filter's doc is scoped to local matching with the outbound REQ path recorded; the settings-filter record citessubscription.ts:8and the unvalidated loader; every ADR 0002 divergence now has a reversal condition that can fire; and expiration rejects0x+10-style signs while stripping U+FEFF, matchingNumber().Round 3 lenses: code, architecture, security (the last because the radix/U+FEFF hunks are its own residues and its surface); performance, ux, ui stand down — the matcher and UTF-16 length paths are byte-identical to round 2.}
chappie-daemon commentedon Oct 19, 2026
Review round 3 closed: security PASS, code PASS, architecture CHANGES. Both code residuals confirmed closed by revert-failure (restoring the strict decoder fails the new NIP-44 test; restoring
>fails both boundary tests), and security re-verified its two residues against NodeNumber()across adjacent forms.One finding remains: moving the boundary to
>= 2^64closed the saturation but also refuses the exact JSON integer18446744073709551615, which is representable asu64and which the capture accepts — the check goes throughas_f64(), which rounds it to 2^64. Round 4 (resume 4 of 5) fixes that to accept the integer form, and gives the ADR's representability divergence its reversal condition.Methodology note worth keeping: a scratch copy built through
scripts/rust.shsharesCARGO_TARGET_DIR=/targetand the/work/nostreammount path, so a scratch revert build can poison the next worktree run (stale artifact, 51/2 reported where the truth was 53/0 until the sources weretouched). Revert tests need that touch or an isolated target dir.}chappie-daemon commentedon Oct 19, 2026
Builder round 4 complete: STATUS: READY, two appended commits (
9e3e171,104dd61), four files, +75/−35. Workspace 546 passed / 0 failed, both clippy passes andfmtclean, release build green, tree clean.The round-3 finding is fixed properly rather than papered over:
validate_kinds/validate_limitnow read the serdeNumber's integer form first, so the exact integeru64::MAXis carried intoKind(u64::MAX)/Some(u64::MAX)instead of being rounded to 2^64 byas_f64(); floats take the comparison path. Tests assert every cell — u64::MAX accepted, 2^64 refused both as a float and as an integer literal, 2^64−2048 and 1e19 accepted — and the exact-integer test fails under round-3 code. ADR 0002 regained the reversal trigger and now states the refused cell exactly, andvalidate_filter's doc matches it.Round 4 lenses: code, architecture, security (the last because the hunk changes a validation guard). Resume 4 of 5.}
chappie-daemon commentedon Oct 19, 2026
Review round 4 closed: security PASS, code PASS, architecture PASS. All three lenses confirmed their round-3 findings by revert-failure and by probing the wire path:
u64::MAXis now carried exactly (Kind(18446744073709551615)), the refused set is unchanged otherwise and is a strict subset of the capture's acceptance, and no truncating cast or comparison overflow is reachable. Against round-3 code the new tests fail 2 of 54; at the head they pass.Hand cross-check before the round line: all three review files read
VERDICT: PASS, matching job.json's slots (ux/ui/performance skipped). Stage is now approve; the factory approver is reading the full job.}chappie-daemon commentedon Oct 19, 2026
Approver: APPROVE, no blocking condition. It re-ran rather than read:
ts_parity8/8 at104dd61, regenerated the fixture from a fresh capture checkout (identical digestc7816541…; both generators exit 2 from the rewrite tree; the capture lockfile behind the symlinked modules is byte-identical), workspace 546/0,fmt+ both clippy passes + release build clean, per-target counts 105/9/31/54/34/17/44/8 all at or above their mocha figures,merge-treeagainstrewrite/mainclean, and C1–C20 re-run itself. It also confirmed no test, guard or expectation was weakened across the job's commits.Two reservations recorded in
decision.md, both inert until NIP-40 retention or a REQ handler is wired, closed by one explicit ECMAScripttrimset:js_numbertrims U+0085 (beyondtrim) andFilter::matchessplits search onsplit_whitespacerather than/\s+/.PR opened: #907 (base
rewrite/main, head104dd61). Merge intorewrite/mainis this manager's, per the charter.}feat(core): implement nostream-core — event model, NIP-01 filters, cr…
chappie-daemon commentedon Oct 19, 2026
Merged. Squash commit
b324ef5e2235—feat(core): implement nostream-core — event model, NIP-01 filters, crypto, NIP semantics (#863)— onrewrite/main. PR: #907.What the job delivered, against the issue's acceptance:
nostream-coreimplemented: 32-byteEventId/Pubkeyand 64-byteSignaturenewtypes with borrowed-hex serde; canonical NIP-01 id + BIP-340 sign/verify; the NIP-01 matcher (31 measured probes, allocation-free prefix compare);validate_filterat the capture's acceptance; NIP-13, NIP-40, NIP-44 v2, reference time, replaceable precedence, and the NIP-77 seam overnegentropy0.5.2.Phase 1 stub, 18NotImplemented, 0todo!):validate_event,adaptive_pow,nip77frames, twelve per-NIP modules.c7816541…), verified independently by the approver.fmtclean; release build green.Review: four rounds, 19 findings raised and all closed (rounds 1–2 code+architecture CHANGES, round 3 architecture, round 4 all PASS). Approver: APPROVE, having re-run the suite, the fixture regeneration, the per-target counts,
merge-treeand C1–C20 itself. Two reservations are recorded in the PR body anddecision.md(ECMAScripttrimset forjs_number; search split on/\s+/), both inert until NIP-40 retention or a REQ handler is wired.Cost — expected $55, actual not measurable. The serving model (
opencode-deepseek-v4.1-flash) has no price configured, so every transcript usage record carriescost: 0andfactory-spanshonestly reported $0.00. What was measured: 19 sessions, 1,572 requests, 2.11M input / 1.43M output tokens, 250.3M cache-read; recon 9m, spec 10m, build 19m46s + three resumes, rounds ≈20/12/8/7m, approve 5m30s. At #862's measured blended rate ($0.0102/request) that is ≈ $16, inside the $40–70 forecast — an inference, not a measurement.Harness findings (not this job's fix set): the model needs a price or an explicit "unpriced" marker so $0.00 is never read as a result; and
factory-spans' lower bound anchored on the firstplanline, excluding this job's own recon, spec and first builder run.