SNAPSHOT_AND_RESTORE.md

Snapshot and restore

XERJ implements a subset of the Elasticsearch snapshot API. A snapshot is a recursive filesystem copy of one or more index directories into a repository directory, plus a manifest.json at the snapshot root. A restore copies those directories back and reopens the indices.

Restore is destructive. It deletes the live index directory before copying the snapshot back. Read Restore replaces index directories before you run one.

Everything below was read out of the source. The handlers are in engine/crates/xerj-api/src/es_compat.rs, the engine work is Engine::create_snapshot and Engine::restore_snapshot in engine/crates/xerj-engine/src/engine.rs, the routes are in engine/crates/xerj-api/src/router.rs, and the authorization rules are in engine/crates/xerj-api/src/authz.rs.

Routes

Method	Path	Handler
PUT	/_snapshot/{repo}	register or replace a repository
GET	/_snapshot/{repo}	read a repository config (* or _all returns all of them)
DELETE	/_snapshot/{repo}	deregister a repository
PUT	/_snapshot/{repo}/{snapshot}	take a snapshot
GET	/_snapshot/{repo}/{snapshot}	read one snapshot's info
POST	/_snapshot/{repo}/{snapshot}/_restore	restore
GET	/_cat/repositories	plain-text name type lines, one per repository, no header row

Those are all the snapshot routes the router registers (router.rs:485-497, router.rs:763). There is no DELETE /_snapshot/{repo}/{snapshot}, no _status, no _verify, no _cleanup, and no snapshot lifecycle management. See What is not supported.

Register a repository

The body you send is stored verbatim under the repository name (es_compat.rs:23138). Only one field is ever read back out: the string at settings.location, which is the filesystem path the snapshot directory is created under (es_compat.rs:23197-23201).

The repository name must not be empty and must not contain .., /, \, or a NUL byte. Violations return HTTP 400 with "type": "mapper_parsing_exception" (es_compat.rs:23086-23098).

The location is bounded. validate_snapshot_path rejects any location that contains a .. path component, and requires the location to canonicalize to a path inside server.data_dir or inside one of the base directories listed in limits.snapshot_repo_allowlist in the TOML config (engine.rs:1971-1997, xerj-common/src/config.rs:960). The allowlist defaults to empty, which means only data_dir is permitted. This is checked when a snapshot is created or restored, not when the repository is registered, so a bad location is accepted by PUT /_snapshot/{repo} and only fails later.

If settings.location is absent the handler falls back to /tmp/xerj-snapshots (es_compat.rs:23201). That fallback is outside the default data_dir and will therefore be refused by the location check, so always set settings.location explicitly.

curl -sS -X PUT localhost:9200/_snapshot/backups \
  -H "Authorization: ApiKey $XERJ_ADMIN_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"type":"fs","settings":{"location":"./data/snapshots"}}'

Response:

{"acknowledged": true}

The repository registry is an in-memory map built empty at engine start (engine.rs:257, engine.rs:405). Nothing reloads it from disk, so repositories have to be registered again after a restart.

Take a snapshot

curl -sS -X PUT localhost:9200/_snapshot/backups/2026-08-04 \
  -H "Authorization: ApiKey $XERJ_ADMIN_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"indices":"logs-*"}'

The body is optional. The only field read is indices (es_compat.rs:23210-23217).

Both spellings of indices

Since rc.10 the create handler parses indices as an array of names or patterns, or as a single string that may itself be comma-separated:

{"indices": ["logs-2026-08", "metrics-*"]}
{"indices": "logs-2026-08,metrics-*"}

Before rc.10 only the array spelling was read. The string spelling fell through to "absent", which means "every index on the node", so {"indices":"logs-*"} captured everything. Both spellings now parse the same way.

Patterns resolve against what you can see

Engine::create_snapshot takes concrete index names, so the handler expands patterns before calling it. It expands them over Engine::index_name_list() (es_compat.rs:23225-23249), which filters through xerj_engine::index_guard::visible (engine.rs:1250-1256). The middleware installs the request's principal as that visibility rule (authz.rs:1272-1273), so the set of indices captured is the set the caller is allowed to see. Before rc.10 the wildcard reached the engine verbatim, matched nothing, and produced an empty snapshot, so {"indices":["*"]} used to back up nothing at all.

The matcher used on this path is glob_match_simple (es_compat.rs:19590-19601). It handles exactly three shapes: * on its own, a trailing wildcard (logs-*), and a leading wildcard (*-2026). Anything else is compared as an exact string, so a mid-pattern wildcard such as logs-*-2026 matches nothing here. _all and * both expand to every visible index.

With no indices at all

If indices is absent, the engine snapshots every open index except its own internal ones. "Internal" here means a name starting with .xerj_ (is_system_index, engine.rs:1925-1927). The reserved agent-memory and second-brain namespace uses the prefix .xerj-memory- with a hyphen (xerj-common/src/types.rs:190), which is not .xerj_, so an unnamed snapshot does include every brain on the node. That is exactly why an unnamed snapshot is superuser-only; see Authorization.

What gets copied, and when

For each selected index the engine flushes the memtable and then copies the index directory recursively: WAL files, segment files, and the schema, settings and es_mapping.json files (engine.rs:1669-1690). The flush happens immediately before that index's own copy, inside the per-index loop, so indices are captured at different instants. There is no cross-index point-in-time consistency, and no lock stops writes arriving during the copy.

An index named in indices that does not exist on the node is skipped (engine.rs:1670-1673), but it is still listed in the manifest, because the manifest records the requested list rather than the copied list (engine.rs:1657-1668, engine.rs:1692-1697).

Response

The operation is synchronous. There is no wait_for_completion on create. The response is {"accepted": true, "snapshot": <manifest>} where the manifest is the same JSON written to manifest.json (es_compat.rs:23256-23260, engine.rs:1692-1712):

{
  "accepted": true,
  "snapshot": {
    "snapshot": "2026-08-04",
    "uuid": "…",
    "version": "8.13.0",
    "indices": ["logs-2026-08"],
    "state": "SUCCESS",
    "start_time_in_millis": 1754265600123,
    "end_time_in_millis": 1754265601047,
    "duration_in_millis": 924,
    "failures": [],
    "shards": {"total": 1, "failed": 0, "successful": 1}
  }
}

The timestamps are real. start_time_in_millis is sampled before the flush and copy work begins and end_time_in_millis after it finishes, so duration_in_millis is the elapsed wall-clock time of the copy (engine.rs:1655, engine.rs:1692, engine.rs:1702).

The shards counts are index counts, not shard counts: all three are derived from the length of the index list (engine.rs:1704-1707). version is the hardcoded string "8.13.0", which is the ES protocol version XERJ reports, not a XERJ version.

The snapshot name is validated the same way the repository name is, and must also not be . or .. (es_compat.rs:23105-23123).

Inspect

curl -sS localhost:9200/_snapshot/backups/2026-08-04 \
  -H "Authorization: ApiKey $XERJ_ADMIN_KEY"

Returns {"snapshots": [<manifest>]}, or 404 index_not_found_exception if either the repository or the snapshot is unknown (es_compat.rs:23265-23283).

This handler reads an in-memory map keyed by "{repo}/{snapshot}", populated when the snapshot was taken in this process (es_compat.rs:23257-23258). After a restart it returns 404 even though the files are still on disk. Restore does not use that map: it reads manifest.json from the repository directory, so restoring a snapshot taken before a restart still works.

There is no GET /_snapshot/{repo}/_all and no wildcard listing of snapshots; the lookup is an exact key match.

Restore replaces index directories

curl -sS -X POST 'localhost:9200/_snapshot/backups/2026-08-04/_restore?wait_for_completion=true' \
  -H "Authorization: ApiKey $XERJ_ADMIN_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"indices":"logs-2026-08"}'

For every index it selects, the restore loop (engine.rs:1857-1892):

1. removes the index from the live index map,
2. calls remove_dir_all on <data_dir>/<index>,
3. recreates the directory and copies the snapshot's copy back,
4. reopens the index and reloads its persisted ES mapping.

Every document written to that index after the snapshot was taken is gone. So is every mapping change. There is no merge mode, no dry run, and no way to restore under a different name (see the rejected options below). If the reopen in step 4 fails, the index is recorded in the engine's failed_indices map and the loop continues to the next index (engine.rs:1895-1898), so the old data is already deleted at that point.

The list of index names in the response is the set the filter selected. An index whose snapshot directory was missing is skipped with a warning (engine.rs:1836-1840) and one whose reopen failed is also skipped, but both still appear in that list. Treat it as "what was attempted", not as proof of success.

Selecting indices to restore

indices accepts the same two spellings as create: an array, or a single string that may be comma-separated (es_compat.rs:23345-23356). Each entry is split on commas again in the engine, and each resulting pattern is matched against the index list in the snapshot's own manifest.json (engine.rs:1765-1806).

The matcher on the restore path is glob_match (engine.rs:2066-2087), a full wildcard matcher supporting * and ? anywhere in the pattern. This is a different matcher from the one the create path uses, so a pattern that selects nothing when taking a snapshot may still select something when restoring one.

• A pattern with no * that matches nothing in the snapshot is an error: HTTP 400, "type": "search_phase_execution_exception", reason [{snapshot}] no index matches [{pattern}] in snapshot (engine.rs:1795-1802). A wildcard that matches nothing is not an error.
• A wildcard never selects a .xerj_* internal index unless the pattern itself starts with . (engine.rs:1786-1793).
• With no indices at all, every index in the manifest except .xerj_* ones is restored (engine.rs:1775-1782).

Index names coming out of the manifest are revalidated with IndexName::new before any filesystem operation, and the destination directory is checked to be inside data_dir both lexically and after canonicalization (engine.rs:1822-1834, engine.rs:1844-1855, engine.rs:1866-1877).

Restore options that are rejected

These five options are refused with HTTP 400, "type": "illegal_argument_exception", reason restore option [{name}] is not supported by this XERJ version (es_compat.rs:23316-23343):

rename_pattern, rename_replacement, feature_states, index_settings, ignore_index_settings

They fail loud rather than being ignored, because ignoring a rename would overwrite the source index instead of creating a copy.

Any other field in the restore body is ignored. The handler reads only indices and the five names above, so include_global_state, include_aliases, partial and friends have no effect and produce no error.

wait_for_completion

wait_for_completion is a query-string parameter and is compared to the exact string "true" (es_compat.rs:23358-23361). wait_for_completion=1 and wait_for_completion=TRUE both read as false.

The restore runs synchronously either way. The flag only picks the response shape (es_compat.rs:23368-23386):

{"accepted": true}

or, with wait_for_completion=true:

{
  "snapshot": {
    "snapshot": "2026-08-04",
    "indices": ["logs-2026-08"],
    "shards": {"total": 1, "failed": 0, "successful": 1}
  }
}

Again, those counts are index counts.

Authorization

Snapshot and restore are the only two routes where an index pattern in a request body is decided by the authorization middleware itself rather than left to the engine's visibility guard (authz.rs:1143-1152). The reason is in the code: create_snapshot walks the index map and restore_snapshot expands against the snapshot manifest and then removes and rewrites index directories, so neither passes through get_index or delete_index, which is where every other body-named target meets the guard.

The rules, as decide and authorize_expression apply them:

• Superuser skips all of it (authz.rs:1291-1293). A principal is superuser when it presents the configured admin key, and also when auth.enabled is false or auth.admin_api_key is empty, which is the point-at-a-folder local posture (auth.rs:214-222). A superuser can back up and restore everything.

• Naming no indices at all is superuser-only. For any non-GET, non-HEAD request under /_snapshot/ with three or more path segments, an empty set of demanded indices is refused (authz.rs:1333-1358). That covers both PUT /_snapshot/{repo}/{snap} with no body and POST /_snapshot/{repo}/{snap}/_restore with no indices, because both cover every index on the node, brains included. The 403 names the resource as <all indices> and the action as create_snapshot or restore_snapshot.

• A pattern that may reach the reserved namespace needs an explicit grant. may_reach_reserved (authz.rs:177-191) judges a pattern by the literal text before its first *: the expression qualifies if that prefix starts with .xerj-memory- or is itself a prefix of .xerj-memory-. So *, _all, .*, .xerj-* and .xerj-memory-alice* all qualify. For those, only a superuser, or a scoped key whose role_descriptors grant that very pattern with the required privilege, is allowed through (authz.rs:1175-1193). An unscoped key is never allowed a reserved-reaching pattern on these two routes.

• Narrower patterns work normally. logs-* does not qualify under may_reach_reserved, so it is not refused and is not subject to the extra check. The ordinary per-tenant backup keeps working.

• Concrete names go through the normal per-index check, resolving aliases first (authz.rs:1194-1202).

• Restore demands write, not read. body_targets records snapshot indices as ReadIndex, and decide upgrades every demand to WriteIndex when the last path segment is _restore (authz.rs:895-905, authz.rs:1342-1347), because a restore overwrites what it names.

A refusal is an ES-shaped 403 naming the resource and the action, with the grant that would fix it (authz.rs:242-262).

Note that "unscoped" is a real category here: a key with no usable role_descriptors keeps its historical reach over ordinary indices but has no privilege at all on .xerj-memory-* (auth.rs:99-106). And an explicit grant is not second-guessed: a scoped key minted with names: ["*"] does satisfy the check above, because * matches every index and that is what the operator asked for. Only a superuser can mint such a key. To keep brains isolated, grant concrete names or a prefix that excludes .xerj-memory-.

The native backup route

POST /v1/admin/backup calls the same Engine::create_snapshot (native.rs:779-815). Its body has three optional fields: repo_path, name and indices (native.rs:770-777). repo_path defaults to <data_dir>/_backups and name defaults to backup-<uuid>. It returns HTTP 201 with the manifest under a manifest key.

Two differences from the ES route are worth knowing:

• It passes indices to the engine unchanged, with no pattern expansion. The engine looks index names up exactly, so a wildcard here matches nothing.
• The authorization middleware classifies /v1/... as a cluster route, not as a snapshot route (authz.rs:473-480, authz.rs:675-677), so the snapshot-specific pattern rule above does not apply to it. It falls through to the general cluster-mutation rule: a scoped key is refused, an unscoped key is allowed (authz.rs:1386-1396).

The repository location check still applies, since it lives inside Engine::create_snapshot.

What is not supported

Confirmed absent from the code, not merely undocumented:

• Deleting a snapshot. The route table registers only PUT and GET on /_snapshot/{repo}/{snapshot} (router.rs:491-493). Removing a snapshot means deleting its directory yourself.
• _status, _verify, _cleanup, _slm. No routes exist for any of them.
• GET /_snapshot with no repository. The route is /_snapshot/:repo; use GET /_snapshot/_all or GET /_snapshot/* to list repositories.
• Listing snapshots in a repository. GET /_snapshot/{repo}/{snapshot} is an exact key lookup; _all and wildcards are not handled.
• Anything but a local filesystem repository. The type field you send is stored and echoed by _cat/repositories but is never acted on. There is no S3, GCS, Azure or HDFS implementation; the engine always does a local recursive directory copy.
• Incremental or deduplicated snapshots. Every snapshot copies the full index directory. There is no shared blob store and no reference counting between snapshots.
• Global cluster state. Only per-index directories and the manifest are written. Index templates, ILM policies, cluster settings, enrich policies, transform pipelines and security keys are not captured, and are not restored.
• Restore rename, per-index settings overrides, and feature states. Rejected with 400, as listed above.
• ignore_unavailable, include_global_state, partial, include_aliases. Not parsed on either verb.
• Persistent repository and snapshot metadata. Both maps are in-memory and are lost on restart.
• Cluster-wide coordination. Both operations act on the local node's data_dir.
• Compression or encryption of the copy. Files are copied as they are on disk.
• Point-in-time consistency across indices, and any guard against concurrent writes or a concurrent restore. See What gets copied, and when.

Related

• docs/ARCHITECTURE.md for where the engine, storage and API layers sit.
• CHANGELOG.md for the rc.10 entry describing the snapshot authorization fix and the indices parsing fix.