Skip to content

Sync: encryption cannot be downgraded, sign-ins stay local, nothing is swallowed - #357

Merged
fylorn merged 2 commits into
feat/sync-backupfrom
feat/sync-fixes
Oct 19, 2026
Merged

fylorn merged 2 commits into
feat/sync-backupfrom
feat/sync-fixes

Conversation

@fylorn

@fylorn fylorn commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Fixes from the review of the sync integration branch (#352), on top of #354 / #356.

Encryption cannot be downgraded

  • Encrypted location (by setting, or an "on" keyring there): unencrypted versions and device records are never read (tw_pack::read_strict); they are skipped and counted.
  • A missing keyring, an older "off" keyring put back, or a keyring that stays unreadable is the keyring_missing state. Encryption is never turned off and the passphrase never dropped because of it.
  • Turning encryption off rewrites the keyring as "off": sealed with the passphrase, plus a plain marker that only devices without a passphrase go by (a device with the passphrase ignores the marker and goes by what the keyring says). Keyrings carry a growing generation, so an old "off" keyring is recognised. Devices keep their passphrase after encryption is turned off.
  • Each device keeps the last keyring it opened. POST /sync/republish puts it back (or makes a new one) and uploads this device's configuration when nothing at the location is readable.
  • After a keyring race on a cloud folder, a device whose own versions no longer open merges two-way, re-uploads, and deletes those versions.

Merging

  • Upstreams signed in on another device stay disabled where there is no sign-in. That disable is local state: it is never uploaded, in rounds, joins (both modes) or after imports. Signing in turns them on as synced.
  • The "nothing changed" shortcut is keyed on the round's starting point plus what the round did, so mid-round edits and arrivals are not swallowed.
  • The base is used only if it is an ancestor of the heads (versions record up to 200 ancestors); otherwise two-way. Pruning keeps the common ancestor of several heads; earlier bases are kept locally.
  • Global sections reset to defaults merge field by field. The invalid-merge fallback applies changes in reference order and repeats.
  • "Keep both" on a gateway key gives the copy a new key and no client. SyncConflict.options lists the allowed choices; "remote" is refused for invalid conflicts.

The location

  • Clearly broken files are skipped at once; files that may still be arriving are waited for 30 minutes. SyncView.skipped counts them.
  • An empty read becomes emptied only after 3 rounds, 10 minutes and the device's own record being gone. Nothing is deleted or turned off. A missing WebDAV folder counts as unreachable. Purge deletes only sync files.
  • iCloud placeholders read as not arrived. Reads are size-limited while streaming. This device's leftover temp files are swept. A request that hits a pooled connection the server just closed is retried once (Apache closes idle connections after 5 s; found in the compatibility run below).
  • S3 prefixes reject .. and a leading /. A plain http:// location cannot use a proxy.

API (still 50)

States keyring_missing, emptied, misconfigured; SyncView.skipped; SyncConflict.options; POST /sync/republish (and twcore sync republish). New codes: sync.keyring_missing, sync.keyring_rolled_back, sync.looks_empty, sync.location_empty (replaces sync.emptied, whose sentence changed), sync.take_not_allowed, sync.too_new_unknown, sync.store.too_large, sync.webdav.root_missing, sync.config.prefix, sync.config.plain_http_proxy.

Tests

  • The review's repro cases are now regression tests: crates/tw-sync/tests/regressions.rs (24) and crates/tw-control/tests/sync_regressions.rs (5).
  • The fake S3 verifies SigV4 signatures with its own implementation, checked against AWS's published examples. A deterministic test covers the stale-connection retry.
  • Local: cargo test --workspace 3822 passed, 0 failed; cargo clippy --workspace --all-targets -D warnings clean; cargo fmt --check clean; scripts/smoke.sh 84 passed, 0 failed.
  • Compatibility run (not in CI): two real twcore processes ran against servers in containers on localhost: MinIO (bitnamilegacy/minio:2025.7.23-debian-12-r5), Apache mod_dav (bytemark/webdav:2.4) and rclone (rclone/rclone:1.69.1 serve webdav). Each run covered encrypted enable, join, changes both ways, a conflict resolved through the endpoints, encryption off and back on, and a purge. 13/13 checks passed on each server.

🤖 Generated with Claude Code

fylorn and others added 2 commits October 19, 2026 16:44
…s swallowed

Fixes from the review of the sync integration branch.

Encryption
- With encryption on (by setting, or a keyring at the location), versions
  and device records that are not encrypted are never read (tw_pack's
  read_strict); they are skipped and counted.
- A missing keyring, an older "off" keyring put back, or one that stays
  unreadable is a state for a person (keyring_missing), never a reason to
  turn encryption off or drop the passphrase. Turning encryption off now
  rewrites the keyring as "off" (sealed with the passphrase, with a plain
  marker for devices that have none), and each keyring carries a growing
  generation so an old one is recognised.
- Each device keeps the last keyring it opened; POST /sync/republish puts it
  back (or makes a new one) and uploads this device's configuration when the
  location has nothing readable.
- A device whose own versions no longer open after a keyring race merges
  two-way, re-uploads with the keyring now there and deletes them.

Merging
- Upstreams signed in on another device (ChatGPT, OAuth tokens) stay
  disabled on a device without the sign-in; that disable is local state,
  never uploaded, in rounds and in both join modes. Signing in turns them on
  as synced.
- The "nothing changed" shortcut is keyed on what the round started from
  plus what it did, so an edit or a version that lands mid-round is synced
  next round. Local content is hashed raw, so a sign-in counts.
- The base is used only when it is an ancestor of the location's heads
  (versions record up to 200 ancestors); otherwise the merge is two-way.
  Pruning keeps the common ancestor of several heads, and earlier bases are
  kept locally to find one that was already pruned.
- Global sections and client prefs reset to defaults merge field by field.
- Invalid-merge fallback applies changes in reference order and repeats until
  nothing more fits.
- Keeping both gateway keys gives the copy a new key and no client.
  Conflicts list their choices (options); "use the other side" is refused for
  invalid conflicts.

The location
- A file that is clearly broken is skipped at once; one that may still be
  arriving is waited for 30 minutes, then skipped. The count is in the view.
- An empty read is checked over several rounds and ten minutes before the
  emptied state; nothing is turned off or deleted automatically. A WebDAV
  folder that is gone is unreachable, not empty. Purging deletes only sync
  files.
- iCloud placeholders read as not yet arrived. Reads are size-limited while
  streaming. Leftover temporary files of this device are swept. A request
  on a pooled connection the server just closed is sent once more.
- S3 prefixes may not contain .. or start with /; a plain http:// address
  cannot go through a proxy.

API (still 50): sync states keyring_missing, emptied, misconfigured;
SyncView.skipped; SyncConflict.options; POST /sync/republish; twcore sync
republish. New codes: sync.keyring_missing, sync.keyring_rolled_back,
sync.looks_empty, sync.location_empty (replaces sync.emptied),
sync.take_not_allowed, sync.too_new_unknown, sync.store.too_large,
sync.webdav.root_missing, sync.config.prefix, sync.config.plain_http_proxy.

The fake S3 in the tests now verifies SigV4 signatures with its own
implementation, checked against AWS's published examples.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The marker in front of an "off" keyring is not authenticated. Only devices
without a passphrase (never encrypted) go by it; a device that has the
passphrase opens the keyring and goes by what it says, so a forged marker
asks for the passphrase instead of turning encryption off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fylorn
fylorn merged commit c772398 into feat/sync-backup Oct 19, 2026
7 checks passed
@fylorn
fylorn deleted the feat/sync-fixes branch October 19, 2026 09:09
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 19, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant