Repository navigation
Sync and backup: config packs, import and export, local backups, multi-device sync - #352
Merged
Merged
Conversation
* tw-pack: configuration pack format and the item model A new crate for the .twconfig configuration pack that export, sharing, local backups and sync versions share: - Format: a ZIP with manifest.json, config.yaml, plugin scripts and their approved copies, and client-prefs.json; optionally sealed with age (scrypt passphrase, or an X25519 recipient for sync). Reading treats the file as untrusted: size limits on the file and on what it unpacks to, only the known paths, valid plugin ids, no duplicates. Errors carry their own message codes (pack.*). Same content writes the same bytes. - Model: a configuration split into items (upstreams, proxies, keys, aliases, groups, routes, price sheets, custom security rules, plugins, the order of upstreams, routes and plugins, each global section, client preferences) and merge units (upstream: conn/models/billing/other; plugin: script/settings/enabled; global settings and preferences field by field). Values are normalized, so comments, layout and written-out defaults are not differences. Local settings (listen, retention, sync, the ChatGPT login and account header, the script of a shipped default plugin) stay out of the snapshot and are kept on materialize. - materialize: lands a target snapshot on a local config by editing it item by item, so untouched items keep their bytes and comments; falls back to a full rewrite when an edit cannot apply, and verifies the result either way. A plugin that can rewrite tool calls is not approved when it arrives enabled from elsewhere. - rename follows every reference; content_id identifies the shareable content (12 hex chars of blake3 over the normalized snapshot). Plugin source splitting needs the plugin runtime, which this crate does not depend on; the core installs it (install_codec), and without it a script is compared whole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * tw-pack: zip 9 (RUSTSEC-2026-0367/0368/0369); a clippy fix for Rust 1.99 The advisories are about extracting archives to disk, which this crate never does, but the patched release is the one to depend on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Export, import and local backups on the control plane and the CLI - Endpoints: /config/items, /config/export, /config/import/preview, /config/import, /backups, /backups/file, /backups/preview, /backups/restore, /client-prefs, /pinned-keys. Import goes item by item (new, same, conflict, error; import, overwrite, rename, skip) and says what each row refers to; restore can take a subset of items, keeps the local settings, ChatGPT sign-ins and pinned keys, and in safe mode brings a version back whole with the current control key. - Local backups replace the change history: every write leaves a full configuration pack (plugins and client preferences included, control key masked) before and after, the last 10 are kept, rotations make none, and the old history files go. /config/history and /config/rollback are removed; ConfigOrigin drops rollback and gains sync, import, restore, repair. - New events gateway_keys_changed and client_prefs_changed. Over a remote connection the client preferences stay in the app. - RemoteBackups lets sync add the versions of the sync location to the list; pinned keys are kept in the data directory. - A disabled upstream may lack its credential, which is how imported ones arrive. A plugin that can rewrite tool calls is never approved by an import or a restore. - CLI: twcore export / import / backup list / backup restore; config history and config rollback are gone. - Control API version 50. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * smoke: the backup endpoints instead of the removed history one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#351) - A pack exported by an older version whose config does not read is repaired in memory by the one-click repair rules (unknown fields removed, unknown values back to the default) before preview and import; ImportPreview and RestorePreview list the repairs. A pack from a newer core is still too new. - A plugin's scope (match.upstreams) refers to the upstreams it names, so an export brings them, skipping one refuses the plugin on import, and a rename follows. Wildcard patterns are not references, and an upstream that is nowhere is not an error for a plugin. - OAuth upstreams: the refresh token rotates (the gateway writes back the new one), so the tokens stay on the device like a ChatGPT sign-in. They are not exported, an overwrite keeps the local ones when the token endpoint is the same, a new one arrives disabled and flagged missing_key, and a restore brings back the tokens of an upstream that is no longer here from the version restored. - control.config_not_repairable becomes control.config_unrepairable and points at restoring a backup. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…conflicts that travel with versions (#350) - tw-config: a local-only `sync` section (device, encryption, passphrase, excluded upstreams, target) with validation; http:// only for this machine and the local network. Documented in the configuration manual. - tw-sync (new): a Store trait (list, get, put_new, put, delete, create_exclusive) with folder, WebDAV and S3 implementations (S3 addressing chosen from the endpoint host); an age X25519 keyring sealed with the passphrase; version naming and retention that keeps every head and its parents; device records; the unit-wise three-way merge with table tests; order merging; conflict records; the sync round (heads merged on their common parent, base from the local base, local backups or the location), joining, encryption on/off and passphrase change, stop and purge. - tw-control: the sync service (10-second settle, upload before exit, 30-minute polls for WebDAV and S3, a folder watcher, checks at start, after wake and with retries; paused in safe mode; unreachable only after two hours); the /sync endpoints and Event::SyncStatus; local writes through the same order as import and restore, with the source device kept on the local backup; RemoteBackups registered while sync is on; pinned keys from the configuration manager; joining with the import preview and apply of tw-pack. - tw-pack: a backup version can record the device it was synced from. - tw-bedrock: sign_service, additive SigV4 for other services (S3). - tw-secret: the sync passphrase is masked in diagnostics. - twcore: sync status, now, enable and disable; the service started and flushed with serve. - CONTROL_API_VERSION 50; message codes sync.*. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… cap 18 (#354) - read_strict(bytes, unseal): with a passphrase or an identity, a pack that is not encrypted is refused (pack.not_encrypted). For a sync location with encryption on, where accepting a plain version would let anyone who can write there change this device's configuration. - The largest scrypt work factor a pack may ask for is now the one packs are written with (18), like the sync keyring; 22 meant about 4 GiB. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…he keys here (#356) - A pack without secrets is compared and overwritten with the secrets already here (upstream keys, header values, AWS keys, OAuth client secrets, proxy passwords, gateway keys): an item that differs only in its secrets is the same, an overwrite no longer blanks the local key, and only a new gateway key is regenerated. - A gateway key imported under another name gets a new value instead of duplicating the one it was copied from. - Restore plans until nothing is missing: an item that stays (a pinned key, an unselected route) keeps what it refers to, shown in the preview as kept for `referenced`; restoring chosen items brings along what they need that is no longer here. An OAuth upstream whose token endpoint changed comes back with the tokens of that version, or disabled when it has none, instead of failing. - A whole restore in safe mode takes listen, the control key, retention and sync from the current text, or from the latest local backup when the text is not YAML, so a sync version or an exported pack can be restored. - Pinned keys are kept per source (PUT /pinned-keys { source, names }), the union is what counts, and a source not heard from in 30 days expires. - Import bodies may carry a full 16 MiB pack in base64; other endpoints keep the default limit. Passphrase decryption and encryption run on a blocking thread. - When the gateway cannot take a configuration that is already written, the plugin files stay as written instead of being rolled back. At start, plugin files that match neither the approved hash nor its copy are put back from a backup. The 64 KiB limit on client preferences applies to imports and restores too. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…s swallowed (#357) * Sync: encryption cannot be downgraded, sign-ins stay local, nothing is swallowed Fixes from the review of the sync integration branch. Encryption - With encryption on (by setting, or a keyring at the location), versions and device records that are not encrypted are never read (tw_pack's read_strict); they are skipped and counted. - A missing keyring, an older "off" keyring put back, or one that stays unreadable is a state for a person (keyring_missing), never a reason to turn encryption off or drop the passphrase. Turning encryption off now rewrites the keyring as "off" (sealed with the passphrase, with a plain marker for devices that have none), and each keyring carries a growing generation so an old one is recognised. - Each device keeps the last keyring it opened; POST /sync/republish puts it back (or makes a new one) and uploads this device's configuration when the location has nothing readable. - A device whose own versions no longer open after a keyring race merges two-way, re-uploads with the keyring now there and deletes them. Merging - Upstreams signed in on another device (ChatGPT, OAuth tokens) stay disabled on a device without the sign-in; that disable is local state, never uploaded, in rounds and in both join modes. Signing in turns them on as synced. - The "nothing changed" shortcut is keyed on what the round started from plus what it did, so an edit or a version that lands mid-round is synced next round. Local content is hashed raw, so a sign-in counts. - The base is used only when it is an ancestor of the location's heads (versions record up to 200 ancestors); otherwise the merge is two-way. Pruning keeps the common ancestor of several heads, and earlier bases are kept locally to find one that was already pruned. - Global sections and client prefs reset to defaults merge field by field. - Invalid-merge fallback applies changes in reference order and repeats until nothing more fits. - Keeping both gateway keys gives the copy a new key and no client. Conflicts list their choices (options); "use the other side" is refused for invalid conflicts. The location - A file that is clearly broken is skipped at once; one that may still be arriving is waited for 30 minutes, then skipped. The count is in the view. - An empty read is checked over several rounds and ten minutes before the emptied state; nothing is turned off or deleted automatically. A WebDAV folder that is gone is unreachable, not empty. Purging deletes only sync files. - iCloud placeholders read as not yet arrived. Reads are size-limited while streaming. Leftover temporary files of this device are swept. A request on a pooled connection the server just closed is sent once more. - S3 prefixes may not contain .. or start with /; a plain http:// address cannot go through a proxy. API (still 50): sync states keyring_missing, emptied, misconfigured; SyncView.skipped; SyncConflict.options; POST /sync/republish; twcore sync republish. New codes: sync.keyring_missing, sync.keyring_rolled_back, sync.looks_empty, sync.location_empty (replaces sync.emptied), sync.take_not_allowed, sync.too_new_unknown, sync.store.too_large, sync.webdav.root_missing, sync.config.prefix, sync.config.plain_http_proxy. The fake S3 in the tests now verifies SigV4 signatures with its own implementation, checked against AWS's published examples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Sync: a device with the passphrase ignores the plain "off" marker The marker in front of an "off" keyring is not authenticated. Only devices without a passphrase (never encrypted) go by it; a device that has the passphrase opens the keyring and goes by what it says, so a forged marker asks for the passphrase instead of turning encryption off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…iCloud placeholders download (#358) Default gateway key on join - Every installation generates a default gateway key. When the key that default_key points to has the same name on both sides and a different value, joining is not a conflict: the joining device takes the sync data's value, in both join modes (a pinned default key is no longer kept over it on use_remote), and GatewayKeysChanged follows so taken-over clients are updated. Decisions given for that item are ignored. - POST /sync/enable/preview lists it in the new SyncJoinPreview.auto ({ item, action: overwrite, reason: default_key }); the row itself stays in rows. auto is filled for initial configurations too. twcore sync enable - Merging asks about each item that differs; the default is "use the sync data", "keep this device's version" says it replaces the other devices' one, and "keep both" is offered where a new name is suggested. Before, pressing Enter skipped every conflict, which pushed this device's values (its default key among them) onto the other devices. - The device name defaults to the computer name: ComputerName (then LocalHostName) on macOS, the host name on Linux, COMPUTERNAME on Windows. iCloud placeholders (macOS 13 and earlier) - Reading the real name of a legacy .name.icloud placeholder does not download it. The folder location now asks iCloud to download it through NSFileManager startDownloadingUbiquitousItemAtURL:error:, called from the system osascript (JavaScript for Automation) with the path as an argument, at most every 5 minutes per file. - A placeholder (an empty read) is "arriving", not "unreadable": it does not count toward the 30 minutes after which unreadable files are skipped, and is skipped only after a day so it cannot hold uploads back forever. The keyring gets the same treatment. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…achine needs (#359) Import - A pack without secrets reuses the keys here only when overwriting an item whose destination is unchanged: same scheme, host and port, AWS region, OAuth token endpoint, and proxy type and address. Otherwise the secret stays blank, the row is flagged missing_key and the upstream comes in disabled. An item imported under another name never takes the keys here. - A proxy whose password is blank is flagged missing_key too. Restore - A route that names a group which is gone resolves to that group: kept when it is being removed, brought back when only the route is chosen. - Proxies used by this machine's own sections (the sync location's proxy) are kept; a whole restore copies the proxy over when the version lacks it. - Restoring a version from the sync location leaves the upstreams this machine does not sync, and what follows them, out of the comparison. - POST /backups/preview takes the chosen items and plans exactly what POST /backups/restore writes; the restore now answers RestoreWritten { version, rows } with the rows it actually applied. - Text that does not read as YAML is no longer kept as a version, and a whole restore takes this machine's sections from the newest readable one. Other - Pinned keys no longer expire; a source replaces its own set, under one lock, with limits on sources, names and lengths (backup.pinned.*). The older file shapes read as the blank source. - Plugin files put back at startup are reloaded by the gateway, and only when neither the script nor its approved copy matches the approved hash. - The 64 KiB preference limit applies to the preferences being written, in every writer; a large one already stored does not block a restore. - A passphrase-encrypted pack that asks for more work than allowed reports pack.too_much_work instead of pack.corrupt. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…nd; CLI explains keep and keep-both (#360) Preferences larger than the limit (they cannot be written here) used to fail the whole apply, every round, until the state read as unreachable. Now: - When the sync data's general preferences are too large, they are held: this device keeps its own, the uploaded versions keep the sync data's (this device's are never pushed over them), and everything else syncs. The same applies when the merged preferences come out too large, and on join. - SyncView.notices carries sync.prefs_too_large. It is not an error and does not count toward unreachable. - Once the sync data's preferences fit again and this device's did not change meanwhile, they land as synced instead of looking like a local edit. - The control plane's apply skips oversized preferences rather than failing. twcore sync enable: "keep this device's version" and "keep both" now say that this device's version is uploaded with the join and replaces the item of that name on the other devices. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…a rename brings (#361) - destination() parses with the url crate, the parser reqwest uses: a backslash separates the path as it does when the request goes out, and hosts are normalized (case, punycode, IPv6/IPv4 forms) with default ports filled in. A trailing dot stays a different name. - Upstream addresses with a backslash or a user name/password before the host are refused (config.base_url_backslash, config.base_url_userinfo). - The outbound proxy an upstream goes through is part of its destination: direct, system, or the proxy's type and address. A different route keeps this machine's keys out. - ImportRow.rename_flags tells what an item brings when imported under another name (it never takes the keys or sign-ins here); the CLI says so. - A whole restore of a version from the sync location brings back the upstreams this machine does not sync, and what follows them, from the configuration here or the newest readable version. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the sync and backup work from the integration branch into main. Each piece was reviewed and merged into
feat/sync-backupon its own:tw-pack: the configuration pack format (.twconfig, ZIP, optional age encryption) and the item model that splits a configuration into items and merge units.ConfigHistoryandConfigRollbackare removed in favour of/backups.Review follow-ups, each merged into the integration branch with regression tests:
CONTROL_API_VERSIONgoes to 50. New message codes:pack.*,import.*,backup.*,sync.*, andcontrol.config_unrepairablereplacescontrol.config_not_repairable.Not released yet; the version bump comes in a separate release PR.
🤖 Generated with Claude Code