Skip to content

feat(storage): repository traits, Postgres backend, and parity harness - #910

Merged
chappie-daemon merged 16 commits into
rewrite/mainfrom
factory/007-storage-lmdb
Oct 20, 2026
Merged

chappie-daemon merged 16 commits into
rewrite/mainfrom
factory/007-storage-lmdb

Conversation

@chappie-daemon

@chappie-daemon chappie-daemon commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

What this lands

The storage seam and the Postgres backend for the rewrite, ported from the measured
repository interfaces — not the LMDB store. That split is the job's own scope
decision (spec.md § Scope decision S1): the seam, the 37 sqlx migrations and the
differential parity harness are one reviewable slice; the LMDB store, migration tool
and benchmark rig are named follow-ups.

  • nostream-storage: nine *Repository traits (51 methods) with no impl in the
    traits file, plus the records/streaming types the port needs.
  • The Postgres backend: the knex schema (37 migrations) re-expressed as sqlx
    migrations, behaviour-identical, kept because the parity oracle needs a reference.
  • The differential parity harness: a capture generator, two Rust test targets
    (migrations, parity), scripts/storage-parity.sh, and fixtures.
  • ADR 0001–0003 and a changeset.

Verification

Every row below was run by the approver, not copied from the build report:

check result
cargo test --workspace --all-features 46 suites, passed=594 failed=0 ignored=2
cargo clippy --workspace --all-targets --all-features -- -D warnings clean
strict clippy (unwrap_used, expect_used, todo, unimplemented, panic) clean
cargo fmt --all --check clean
cargo build --workspace --all-features --release succeeds
migrations target 37 _sqlx_migrations rows; second run is a no-op; catalogue equals schema.json
parity target empty diff over 117 entries; 52 ops = all 51 trait methods + one probe
Cucumber suite against the sqlx-migrated schema 118 scenarios (118 passed), 587 steps
scripts/storage-parity.sh exits 0, trap removes both containers

The parity fixture routes each op through the client inside run_transaction, and its
transaction groups cover both sides: one commits an enqueue ({failed: false}), one
records {failed: true, at: 2}, and one enqueues a probe, fails on a constraint and is
followed by a claim that shows only the committed row — so a partial write is observable
rather than assumed.

Review follow-ups

Greptile's three P2s are fixed on the branch (de5f3ef, 12539d4):

  1. postgres/metadata.rs bound FieldUpdate::Set as NULLInt whenever the count was negative, which is the sentinel reserved for Clear; Set(-5) silently became unlimited. The integer is bound as given and chk_remaining_uses_non_negative refuses it.
  2. tests/parity.rs enqueued through the backend even inside run_transaction, so PostgresTransaction::enqueue was never exercised. Enqueue goes through the client now, and reverting that routing makes the new rollback assertion fail.
  3. scripts/storage-parity.sh spliced both passwords raw into the admin URL and the Cucumber URLs; they are percent-encoded for the URLs, with the raw value still passed to the containers.

Why it needs a human

The approver's verdict was ESCALATE on money-access-identity (also
schema-or-dependency and deployment-config): it implements confirm_invoice,
charge_user, balances and admission, and adds 37 migrations, three workspace
dependencies and a CI service. Nothing in the checks failed — it is an
unattended-landing refusal. Merged on the repository owner's explicit instruction.

The PR run is the first CI run on this branch; criterion 18 of the approval is
"one CI run on the PR" and it is verified by this PR's checks, not before it.

… the credential is not echoed

The 127.0.0.1 default is right for CI and for a host run and wrong for this sandbox,
where the cargo tests run inside the builder container whose loopback is not the daemon
host's. Preflight now refuses a loopback bind against a remote daemon with one line
naming PARITY_BIND and PARITY_HOST, probes the dialled address with a three-second
timeout before the first container starts, and runs a forced connection-failure test
that fails the harness if the credential reaches the log.
…ck checks; correct the record and add the language switch
@changeset-bot

changeset-bot Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 12539d4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@socket-security

socket-security Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedcargo/​chrono@​0.4.459910093100100
Addedcargo/​sha2@​0.10.910010093100100
Addedcargo/​sqlx@​0.9.096100100100100

View full report

…enqueue

FieldUpdate::Set bound a negative count as NULLInt, the sentinel reserved
for Clear, so Set(-5) silently became unlimited. Bind the integer and let
chk_remaining_uses_non_negative refuse it.

The parity fixture enqueued through the backend even inside a transaction,
so PostgresTransaction::enqueue was never exercised and the row survived a
rollback. Enqueue through the client, and add a transaction that enqueues,
fails on the negative invite count, and is followed by a claim that proves
the row rolled back.
PARITY_PG_PASSWORD and PARITY_REDIS_PASSWORD were spliced raw into the
admin URL and the Cucumber URLs, so a password containing @, :, /, ? or #
truncated the userinfo. Encode for the URL and keep passing the raw value
to the containers.
@chappie-daemon chappie-daemon changed the title Storage seam + Postgres backend, with the differential parity harness (#864) feat(storage): repository traits, Postgres backend, and parity harness Oct 20, 2026
@coveralls

coveralls commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Coverage Report for CI Build 39352423935

Coverage remained the same at 73.973%

Details

  • Coverage remained the same as the base build.
  • Patch coverage: No coverable lines changed in this PR.
  • No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 10291
Covered Lines: 7992
Line Coverage: 77.66%
Relevant Branches: 5362
Covered Branches: 3587
Branch Coverage: 66.9%
Branches in Coverage %: Yes
Coverage Strength: 869.65 hits per line

💛 - Coveralls

@greptile-apps

greptile-apps Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Critical impact] The Postgres backend reaches stored relay data, payment balances, and user admission.

Enable PostgreSQL TLS support before merging; the transaction test helper also needs a small follow-up.

Findings

  1. P1 TLS database connections fail ▶
  2. P2 Transaction tests lose message IDs ▶

Summary

Adds the storage interfaces and a PostgreSQL implementation, with a recorded TypeScript reference for later backends.

  • Rust callers can use shared storage interfaces with a Postgres backend.
  • Postgres stores and streams relay events with their storage-only fields.
  • Postgres records invoices, balances, and user admission changes.
  • Postgres stores verification, invite, job, and report records.
  • Postgres queues notifications and records their delivery results.
  • A parity harness compares the Postgres backend with captured TypeScript behavior.

Intentional or deferred choices recorded by PR author chappie-daemon: LMDB, the migration tool, and benchmarks are follow-ups; Cucumber remains a local check. ADR 0003 also records mixed search/non-search query failure, partial commits across large event batches, and reduced comparison of generated timestamps and storage metadata as known limits.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Capture[TypeScript capture] --> Fixtures[Recorded results]
    Fixtures --> Replay[Rust parity replay]
    Replay --> Traits[Repository traits]
    Traits --> Backend[PostgresBackend]
    Traits --> Transaction[PostgresTransaction]
    Backend --> Master[(PostgreSQL master)]
    Backend --> Reader[(Optional read replica)]
    Transaction --> Master
    Migrations[37 embedded migrations] --> Master
    Fixtures --> Compare[Compare results]
    Replay --> Compare
Loading

Reviews (1) · Last reviewed commit: "fix(scripts): percent-encode parity cred..." · Reviewed by Greptile

Comment thread Cargo.toml
serde_json = "1"
serde_yaml = "0.9"
sha2 = "0.11.1"
sqlx = { version = "0.9", default-features = false, features = ["runtime-tokio", "postgres", "migrate", "macros", "chrono", "json", "tls-none"] }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 TLS database connections fail

If PostgreSQL requires TLS, the new backend cannot connect. The sqlx dependency disables default features and selects tls-none, leaving no TLS provider for URLs with sslmode=require or sslmode=verify-full. Enable a TLS provider and add a connection test that requires TLS.

Comment on lines +955 to +960
for (index, nested) in ops.iter().enumerate() {
match run_op(
backend,
Client::Transaction(&transaction),
nested,
&mut HashMap::new(),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Transaction tests lose message IDs

run_transaction gives every nested operation a fresh refs map. An outbox.enqueue followed by outbox.mark_delivered or outbox.mark_failed in the same group therefore loses the message ID and fails before reaching the database. This limits the transaction flows the parity harness can test. Keep one map for the group, preserve committed references for later operations, and discard references from rolled-back work.

@chappie-daemon
chappie-daemon merged commit 827558a into rewrite/main Oct 20, 2026
18 of 20 checks passed
@chappie-daemon
chappie-daemon deleted the factory/007-storage-lmdb branch October 20, 2026 01:22
@chappie-daemon

Copy link
Copy Markdown
Collaborator Author

Both findings are being fixed on the branch; neither is dismissed.

P1 (TLS). Correct, and the wider point matters: managed Postgres rejects plaintext, so tls-none makes the backend unusable where it is most likely to run. The sqlx feature moves to tls-rustls-ring-webpki (webpki roots, no system OpenSSL). The suite and the parity script will run again afterwards. One honest limitation to state: the parity script brings Postgres up without TLS, so no test here exercises a TLS-requiring server — filed as #911 for a TLS parity leg with a negative control, rather than claiming coverage this PR does not have.

P2 (per-op refs). Correct. run_transaction allocates a fresh map for each nested op while the top-level script threads one across ops, so a sequential outbox write inside a transaction could not reference its own message id — the harness could not express the case. The caller's refs is threaded through, and the fixture gains a transaction that enqueues and marks delivered in one go, with the negative control recorded.

@chappie-daemon

Copy link
Copy Markdown
Collaborator Author

Post-merge record, so the two late findings are not lost behind a green rollup.

The squash landed at 827558a at 01:22:08Z. Greptile posted its review minutes later; neither finding was a required context, so neither gated the merge. I verified both against the merged tree before carrying them forward:

Both are follow-ups on the merge, not accepted risks: the branch was reviewed as a storage seam plus a Postgres backend for the parity oracle, and a backend that cannot speak TLS to the database it targets is a defect regardless of that scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants