ShardLift

ShardLift is a Linux-only crash-consistency lab for sharded training state. Its killer-demo interrupts a deterministic Fashion-MNIST checkpoint transaction with a real SIGKILL, recovers from immutable local records, offline-reshards the selected state from four workers to three (W4 -> W3), continues training, and records bounded state and evaluation evidence beside an uninterrupted reference.

Evidence-derived six-frame workflow from the two retained local run records: the recorded real SIGKILL, commit-record installation boundary, fresh recovery choice, batch replay or no replay, offline W4 -> W3 resharding, and final checkpoint. It plays once. It is not an OS capture, a fresh run, canonical release evidence, or a benchmark.

Reader map

Reader need	Where to look
Setup	Prerequisites and setup and Run and inspect
Core workflow	What a successful killer-demo run is configured to verify
Key capabilities	State continuity and the evidence bundle
Hard technical decisions	Implemented architecture
Results	Recorded local outcomes
Reproducibility	Tests and reproducibility
Explicit claim boundaries	Evidence status and Limitations and nonclaims

Evidence status

Local evidence, not release evidence. The captured and mixed-evidence visuals use fields from two retained runs recorded on Python 3.12.3 with PyTorch 2.13.0+cpu at clean runtime source revision eb566e6cc593bebed84af13ca6e4dd03eef56aca. Both bundles pass the public structural verifier. They are noncanonical local observations; the revision-specific P0 release gate remains pending.

shardlift verify-bundle checks canonical bytes, stable local file identities, SHA-256 digests, and cross-record bindings. It does not establish authenticity, freshness, rollback resistance, installed-package provenance, or independently recompute the ML results. The local capture preserves those nonclaims as data.

The killer-demo options --repository and --expected-revision validate and record a clean physical checkout. They do not, by themselves, bind the bytes of the installed shardlift package to that checkout. The canonical CI procedure adds the wheel and runtime checks needed for that stronger boundary.

Mixed workflow view: the renderer's source-derived capture/check sequence is populated with the retained revision and runtime fields, then checked byte-for-byte. Open the full-size SVG.

Reader-facing evidence:

• normalized terminal transcript;
• local evidence summary and explicit claim boundary;
• sanitized two-run capture;
• exact file-fsync verify-bundle JSON;
• exact rename verify-bundle JSON; and
• generated inventory and SHA-256 manifest; and
• adopted PNG/GIF manifest, which binds all seven source inputs, both binary outputs, raster-pixel digests, and all six GIF fact records.

Run and inspect

Start from a clean checkout, use short absolute paths outside the repository, and reserve a fresh run directory:

REPOSITORY="$(git rev-parse --show-toplevel)"
REPOSITORY="$(cd "$REPOSITORY" && pwd -P)"
EXPECTED_REVISION="$(git -C "$REPOSITORY" rev-parse --verify 'HEAD^{commit}')"
test -z "$(git -C "$REPOSITORY" status --porcelain=v1 \
  --untracked-files=normal --ignore-submodules=none)"

CACHE_PARENT="${XDG_CACHE_HOME:-$HOME/.cache}/shardlift"
install -d -m 700 "$CACHE_PARENT"
CACHE_PARENT="$(cd "$CACHE_PARENT" && pwd -P)"
CACHE_DIRECTORY="$CACHE_PARENT/fashion-mnist"
install -d -m 700 "$CACHE_DIRECTORY"

RUN_DIRECTORY="/tmp/shardlift-fsync-${EXPECTED_REVISION:0:12}"
test ! -e "$RUN_DIRECTORY"

shardlift killer-demo \
  --run-directory "$RUN_DIRECTORY" \
  --cache-directory "$CACHE_DIRECTORY" \
  --barrier commit-after-file-fsync \
  --repository "$REPOSITORY" \
  --expected-revision "$EXPECTED_REVISION"

The command prints one JSON record only after publishing the evidence bundle. Exercise the other authority outcome with another fresh short run path and --barrier commit-after-rename. Failed runs are intentionally retained for inspection and cannot be resumed in place.

Structurally verify either completed bundle without mutating it:

shardlift verify-bundle --bundle-directory "$RUN_DIRECTORY"

Evidence-derived raster of the committed path-free verify-bundle transcript. It shows structural status, installation and recovery choice, replay count, final W3 checkpoint, and recorded exact-logit equality for each retained run. It is not a terminal screenshot and does not independently recompute ML or installed-package provenance.

Check that the committed evidence, generated SVGs, and adopted PNG/GIF still match their source bytes and production renderers, then run the separate standard-library raster auditor:

python -B tools/render_readme_visuals.py check
python -I -B tools/render_readme_media.py check \
  --repository "$REPOSITORY" \
  --media-directory "$REPOSITORY/docs/visuals/media"
python -I -B scripts/verify_readme_media.py \
  --repository "$REPOSITORY" \
  --directory "$REPOSITORY/docs/visuals/media"

Captured CLI evidence: path-free fields from the exact public verify-bundle output, including the structural scope and final checkpoint for each run. Open the full-size SVG.

Recorded local outcomes

The two retained runs exercised opposite sides of immutable commit installation:

Barrier	Recorded transcript	Candidate installed	Fresh recovery selected	Replay	Final W3 checkpoint
commit-after-file-fsync	342 events; event 341 unacknowledged	no	generation 0, step 1 (current)	1 batch	generation 2, step 2
commit-after-rename	343 events; event 342 unacknowledged	yes	generation 1, step 2 (stale)	0 batches	generation 3, step 3

In each retained audit, the publisher received signal 9, returned -9, and was reaped; the fresh recovery child returned 0 and was reaped. Five sequential worker groups accounted for 19 direct worker exits (4 + 4 + 4 + 4 + 3). Both records contain a complete 10,000-sample evaluation and record exact equality for the frozen 128-by-10 FP32 logit probe. These are captured fields, not independently reproduced conclusions.

Captured outcomes: candidate installation, selected generation, replay count, and final W3 step are recorded alongside one another for each run; this view does not infer causality. Open the full-size SVG.

Captured fault boundary: each exact acknowledged prefix ends at the unacknowledged target; the target callout is widened for legibility and followed by a recorded, reaped SIGKILL. Open the full-size SVG.

State continuity

ShardLift gives model parameters, AdamW slots, optimizer step, and the next-unconsumed data cursor stable logical identities independent of rank ownership. Offline resharding changes placement, not the projected logical state. The retained audits record three complementary checks:

• checkpoint lineage and the selected authority at each barrier;
• equal projected logical-state roots across W4 -> W3 resharding and continuation; and
• equal topology-normalized data-ledger trace digests for baseline, recovery, and continued paths.

Captured lineage: the file-fsync run selects genesis around an uninstalled candidate; the rename run selects the installed candidate before reshard and continuation. Open the full-size SVG.

Captured bounded projection: logical-root and fixed-probe digest equality as stored in the audits. This does not recompute ML provenance. Open the full-size SVG.

Captured ledger continuity: normalized trace digest prefixes match within each run, with the recorded committed and replayed batch counts. Open the full-size SVG.

Implemented architecture

The long-lived orchestrator owns lifecycle and evidence authority. It launches five coordinator sessions sequentially; each coordinator directly owns a fixed worker group. Checkpoint publication and fresh recovery run in isolated children, so the parent cannot mistake its own cached state for a recovery result.

Mixed topology view: recorded worker/process outcomes are mapped onto the source-derived parent, coordinator, publisher, and recovery relationships. Open the full-size SVG.

The checkpoint store separates content from authority. Content-addressed objects and manifests may exist without making a generation selectable; only a valid immutable record installed under commits/ does that. refs/LATEST is a rebuildable hint, never the source of truth. Recovery scans admissible commit records and their dependencies under the implemented local fail-stop model.

Source-derived protocol: writer order and both implemented crash targets from RFC 0001. This diagram explains code paths; it is not a claim that every retained run reached every later step. Open the full-size SVG.

PyTorch supplies CPU tensor math and autograd. ShardLift implements the local process protocol, coordinator-mediated collective, stable logical chunk identities, sharded AdamW state, topology-independent data cursor, canonical codecs, checkpoint transaction, recovery selection, offline resharding, and evidence publication. It does not use torch.distributed as its correctness boundary.

See the architecture and authority guide for process credentials, checkpoint authority, validation order, and threat boundaries.

Evidence bundle

A completed run has four adjacent committed records:

1. reference-evaluation.json;
2. recovered-evaluation.json;
3. recovery-evidence.json; and
4. audit.json, the bundle commit marker.

The publisher protocol installs audit.json last. The structural verifier decodes closed schemas, checks canonical re-encoding, revalidates local file identity, recomputes digests, and follows cross-record bindings; it does not reconstruct historical syscall order. Presence of a structurally valid audit.json means the bundle is internally complete under the stated trust model, not that it has an external signature or freshness oracle.

Mixed bundle view: recorded canonical artifact sizes and digests are mapped onto the source-derived publisher order; audit.json is the commit marker. The structural verifier does not reconstruct historical install ordering. Open the full-size SVG.

Maintainers can regenerate visuals only from two retained run directories at the same clean runtime revision:

python -B tools/render_readme_visuals.py capture \
  --file-fsync-run /absolute/private/file-fsync-run \
  --rename-run /absolute/private/rename-run \
  --expected-runtime-revision eb566e6cc593bebed84af13ca6e4dd03eef56aca

python -B tools/render_readme_visuals.py check

The capture command verifies both bundles first, stores exact path-free CLI output, binds the runtime source inventory to the named Git object, rejects unsafe or stale inventories, and publishes the manifest last. The expected revision must exactly match both retained audits. It may be an ancestor of the current documentation commit, but every tracked runtime file must still be byte-identical to that Git object. Private run paths, hostnames, PIDs, UIDs, GIDs, timestamps, and invocation identifiers are not copied into the checked-in reader evidence.

What a successful killer-demo run is configured to verify

One invocation is configured to:

1. validate a clean physical Git checkout and full 40-character revision when the public source-binding options are supplied;
2. acquire and SHA-256-check the four pinned Fashion-MNIST resources;
3. run five sequential coordinator-supervised sessions and collect all 19 worker exits;
4. publish matching committed genesis state to independent reference and crash stores;
5. kill an isolated checkpoint publisher at one named barrier with a real SIGKILL;
6. make a fresh process recover by scanning immutable commit records rather than trusting refs/LATEST;
7. continue the selected W4 state and compare it with the matching uninterrupted reference;
8. independently reshard the selected checkpoint from W4 to W3, continue the same next batch, and compare projected logical state and ledger history;
9. evaluate the complete pinned 10,000-sample test split and retain exact FP32 bytes for the frozen 128-sample logit probe; and
10. install the four-file evidence bundle with audit.json last.

The two barriers answer different local fail-stop questions:

CLI value	Kill point	Configured recovery result
commit-after-file-fsync	staged commit record is written and file-fsynced but not installed in commits/	candidate is not selectable; recover the prior committed generation and replay the next batch
commit-after-rename	immutable commit record is visible in commits/, before commits-directory fsync and before LATEST advances	fresh recovery selects the installed candidate; no batch replay

The rename observation concerns a live local filesystem after process failure. It is not a power-cut durability result.

Prerequisites and setup

The torch-free core package supports Python 3.11 or newer. The killer-demo additionally requires:

• Linux /proc, pidfds, Unix-domain sockets, and POSIX descriptor and signal behavior;
• the CPU build of PyTorch 2.13.0;
• Git and a clean physical checkout with a known full commit SHA;
• a short absolute fresh run path and an existing owned cache directory;
• network access for first acquisition unless all four exact compressed Fashion-MNIST resources are already in the verified cache; and
• enough time and memory for five sequential sessions and two full test-split evaluations.

Long paths can exceed the Unix-socket pathname limit. Keep the dataset cache and run directories outside the source tree.

python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
python -m pip install \
  --index-url https://download.pytorch.org/whl/cpu \
  "torch==2.13.0+cpu"
shardlift --version

ShardLift is not documented as a published package yet. For a wheel-oriented install:

python -m pip install "build>=1.2"
python -m build --sdist --wheel --outdir dist

python3 -m venv .run-venv
. .run-venv/bin/activate
python -m pip install --no-deps dist/shardlift-0.1.0.dev0-py3-none-any.whl
python -m pip install \
  --index-url https://download.pytorch.org/whl/cpu \
  "torch==2.13.0+cpu"
shardlift --version

Retain the wheel and its checksum for provenance-sensitive work. An editable install is not a substitute for the canonical wheel boundary.

Tests and reproducibility

Lane	Command	Scope
Fast deterministic	python -m pytest -q -m "not process and not ml and not slow"	codecs, schemas, validation, placement, and pure checkpoint logic
Local non-slow	python -m pytest -q -m "not slow"	normal unit, bounded process, and ML handoff lane
Format and lint	python -m ruff format --check . and the scoped command in CONTRIBUTING.md	mechanical style and static lint; RUF036 is waived only for three evidence-bound files
Strict typing	python -m mypy src tests	strict package and test typing
Branch coverage	python -m pytest -q --cov=shardlift --cov-report=term-missing	full branch-aware suite with the configured 95% threshold
Visual freshness	python -B tools/render_readme_visuals.py check	exact evidence inventory, runtime-source binding, deterministic render, and manifest
Adopted PNG/GIF	python -I -B tools/render_readme_media.py check --repository . --media-directory docs/visuals/media plus python -I -B scripts/verify_readme_media.py --repository . --directory docs/visuals/media	production byte replay plus independent PNG/GIF, pixel, manifest, and frame-fact verification
Wheel	python -m build --sdist --wheel --outdir dist	distribution boundary

Also run:

python -m mypy --strict scripts/verify_canonical_evidence.py
python -m mypy --strict tools/render_readme_visuals.py
git diff --check

CI covers the non-slow surface on Python 3.11, 3.12, and 3.13 and its core quality lane on Python 3.14. The narrower canonical evidence job is configured for CPython 3.14.4, Linux x86-64, PyTorch 2.13.0+cpu, a hash-locked wheelhouse, and local ext4. A configured workflow is not a retained release result; see the P0 evidence plan.

Limitations and nonclaims

• ShardLift is 0.1.0.dev0, pre-alpha, and not a production training framework.
• W4 -> W3 is offline checkpoint resharding, not live elasticity.
• A real SIGKILL exercises fail-stop process recovery, not power-loss durability.
• The evidence boundary is one trusted local Linux host and local-filesystem semantics; it does not cover multi-host training, NFS, object stores, Windows, or macOS.
• CPU eager execution is covered. GPU, CUDA, NCCL, FSDP, and distributed-checkpoint interoperability are not.
• SHA-256 binds bytes and detects corruption. It is not a signature, authenticity mechanism, freshness oracle, or external anti-rollback anchor.
• Workers, coordinator, kernel, source checkout, and invoking user are trusted. Byzantine workers and hostile same-UID or root actors are outside the model.
• Local visual verification does not bind installed package bytes and does not independently reconstruct dataset, checkpoint, or ML provenance.
• Bitwise observations are scoped to the recorded environment; no cross-release or cross-platform bitwise reproducibility is claimed.
• Fashion-MNIST is a controlled correctness fixture. No model-quality, convergence, throughput, latency, scalability, or memory-saving result is claimed.
• Accuracy, loss, and recorded phase timings are audit context, not benchmark results.
• Replay is deterministic training replay, not exactly-once handling of external side effects.

The normative technical boundary is RFC 0001. Security assumptions and private reporting guidance are in SECURITY.md.

Status and roadmap

Current status is pre-alpha development at 0.1.0.dev0. The implementation, test lanes, wheel checks, visual-evidence freshness checks, and canonical-evidence workflow are present; no stable release or independently validated result is claimed.

Near-term work:

1. retain a clean, revision-specific two-barrier run from the pinned canonical environment;
2. complete the pending P0 acceptance procedure and review its uploaded inventories;
3. keep the CLI, audit schema, RFC, verifier, and reproducible visuals synchronized; and
4. extend fault and corruption coverage before specifying broader platforms or backends.

License

Apache License 2.0.