Multi-device sync: folder, WebDAV and S3 locations, three-way merge, conflicts that travel with versions

[fylorn]

Multi-device configuration sync (the sync half of sync and backup).

What it does

Several devices (desktop cores and remote cores) keep one configuration in step through a sync location the user picks: a folder carried by a cloud drive, a WebDAV server, or S3-compatible storage.

• Append-only location. Versions are named <seq>-<device>.twconfig and never overwritten; only the keyring is created exclusively (If-None-Match: * / hard link), and device records are written by their own device. Folder and WebDAV writes go to a dot-named temporary file first, then rename/MOVE. Files that cannot be read yet (cloud placeholders, partial uploads, wrong key) count as not arrived. Retention keeps every head and their parents, so parallel or late versions survive until merged.
• Three-way merge by merge unit (tw-sync::merge), on tw_pack::model::Snapshot: upstreams merge by conn/models/billing/other, plugins by script/settings/enabled, global settings and client prefs by field, everything else whole. Table-driven tests cover every row of the plan's merge table plus no-base, held items and held units. Orders (upstreams, routes, plugins) merge content and relative order separately, so two devices adding different items is not a conflict.
• Base lookup: the local base (sync/base.twconfig), then local backups, then the location; without a base it compares two ways and keeps one-sided objects.
• Conflicts travel with the versions: each record names the shared side (in the version) and the other side; every device sees the same conflict, keeps its own content meanwhile, and resolving on one device clears it everywhere. Take local / remote / both (name (device)), with deletions, a gateway key in use here (key_in_use) and merged results that fail validation (invalid) handled as conflicts.
• Encryption: an age X25519 version key sealed with the passphrase (scrypt, fixed work factor) as keyring; versions and device records are encrypted to it. Enable / disable / change passphrase; other devices go locked until the new passphrase is entered, and follow when encryption is turned off. Unencrypted versions carry a SHA-256 of their content; plugins that arrive or change through unencrypted sync stay disabled on the receiving device, and address changes are flagged on the upstream.
• Excluded upstreams stay local, together with the routes, aliases and groups that refer to them.
• Scheduling (tw-control::sync): local changes settle for 10 s and go up as one version; upload before exit; WebDAV and S3 every 30 minutes, folders through a watcher; check at start, after wake and with 1/2/5/10-minute retries after failures. Safe mode pauses sync. unreachable only after 2 hours of failures.
• Version too new: a version this core cannot read pauses sync (no download, no upload) and the device record says so.

Contract

CONTROL_API_VERSION 50. New endpoints under /sync (status, test, enable/preview, enable, settings, now, conflicts, encryption, unlock, disable), Event::SyncStatus { id, view, at_ms }, ProviderView.sync_excluded and ProviderView.sync_url_changed, ConfigOrigin::Sync. A local-only sync section in config.yaml with validation (http:// only for this machine and the local network), documented in the configuration manual. New message codes are sync.*. CLI: twcore sync status | now | enable | disable [--purge].

Other changes

• tw-bedrock: sign_service, an additive SigV4 entry point for other services (S3 single-encodes and does not normalize the path); tested against an independent implementation.
• tw-secret: passphrase is masked in diagnostics like the other secret fields.
• S3 addressing is chosen from the endpoint host (virtual-host for amazonaws.com, aliyuncs.com, myqcloud.com; path style otherwise).

Tests

Store conformance against a folder and in-process fake WebDAV and S3 servers (including servers without MOVE or conditional writes); two devices alternating edits, simultaneous uploads merged on their common parent, a late arrival, a conflict handled once on both devices, keep both, delete against edit, coming back after the base was deleted, encryption on/off and passphrase change, one keyring only, a newer core pausing, excluded upstreams, a key in use, a purged location, and the same flow over WebDAV and S3. Control-plane tests drive two full control planes through the endpoints, including the 10-second settle and the folder watcher.

Integration with the pack and backup half: local writes go through the same order as import and restore (plugin files, client preferences, then the configuration, under the plugin edit lock); the local backup of an applied version records the device it came from (tw_pack::backup::note_device, shown as BackupEntry.device); RemoteBackups is registered while sync is on (sync-<content_id> for versions only at the location); pinned keys come from ConfigManager::pinned_keys(); the merge base can come from a local backup with the same content id; joining with a configuration uses the import preview and apply of tw-pack, except that plugins keep the sync data's on/off state.

🤖 Generated with Claude Code