[Ivar (aɪvɑr) (ivarprudnikov)]

The context here is our instances limited to 16G memory and edge case ledger with snapshot approaching 2G. As we know this is not great because DR needs multiples of snapshot memory which can end up in a situation when it gets OOM killed in the process of a DR.

As a dutiful citizen of the internet I ran some checks locally and let the agent go through the existing code to find some low hanging fruit to improve the state of affairs. It came out with some elaborate multi phase improvement but I want to start with this proposal that focuses on the sane parts of not copying data multiple times. It looks legit and I went through the code, local tests show it would use close to 50% less memory during DR and snapshot generation times. The issue, as you can imagine, is that I might be blind to the edge cases.

Summary

DR from a snapshot of size S peaked at about 5.4 x S (5.3 GiB for a 1 GiB snapshot), mostly from redundant full-size copies of the snapshot. Remove them, without any ledger or snapshot format change: the peak falls to about 2.4 x S (2.4 GiB for 1 GiB, 4.6 GiB for 1.9 GiB), and a primary generating a periodic snapshot falls from about 4.5 x S to 2.5 x S.

Structured changes

Step	Change	What it removes
1a	Decrypt and parse without copies	A copy of the ciphertext, a copy of the public domain, a second plaintext buffer, and a copy of each table
1b	Generate a snapshot into one exactly-sized buffer	A temporary copy of each table, growing per-domain buffers and copies of both domains: about 2-3 x S
1c	Decrypt the private domain in place	The remaining plaintext buffer, the size of the private domain
1d	Reuse public state in the private recovery store	A second copy of the deserialised public state

7. Files changed

File	Steps	Change
include/ccf/crypto/symmetric_key.h	1a	decrypt() into a caller-provided span on KeyAesGcm and KeyAesGcm::Context
src/crypto/symmetric_key.cpp	1a	Default implementations of the span decrypt()
src/crypto/openssl/symmetric_key.h, .cpp	1a, 1c	Decrypt directly into the output, in place if requested; erase the output on failure or exception
src/kv/kv_types.h	1a, 1b, 1c	Interfaces: span decrypt(), decrypt_in_place(), serialise_raw_in_place(), deserialise_raw_span()
src/kv/encryptor.h	1a, 1c	TxEncryptor span decrypt() and decrypt_in_place()
src/kv/null_encryptor.h	1a, 1b	Span decrypt() and in-place encryption, for tests
src/kv/raw_serialise.h	1a, 1b	RawReader::read_next_span(), SpanWriter, in-place appends and buffer views for all writers
src/kv/generic_serialise_wrapper.h	1a, 1b, 1c	Span decryption, constructor taking writers, finalise_in_place(), domain sizes, copy-free get_raw_data(), init_in_place()
src/kv/kv_serialiser.h	1b	SpanKvStoreSerialiser
src/kv/snapshot.h	1b	Size pass, then serialisation into one exactly-sized output
src/kv/untyped_map.h	1a, 1b, 1d	Tables written in place and parsed from spans; helpers to deserialise and install a captured state
src/kv/store.h	1c, 1d	deserialise_snapshot_in_place(), SnapshotPublicMaps, capture and checked reuse
src/node/snapshot_serdes.h	1c, 1d	Node-level wrappers for capture and in-place deserialisation
src/node/node_state.h	1c, 1d	Recovery and join use them; the startup snapshot is consumed by private recovery
src/kv/test/kv_snapshot.cpp	1b, 1c, 1d	The two new tests

Compatibility

• The ledger format, snapshot format, configuration and protocol are unchanged, and generated snapshots are byte-identical to main's, so services mixing versions are unaffected.
• include/ccf/crypto/symmetric_key.h is a public header: KeyAesGcm and KeyAesGcm::Context gain a virtual decrypt() overload with a default implementation.
  Existing implementations still compile; as with any change to a virtual interface, code built against the old header must be rebuilt.
• Private recovery consumes the startup snapshot, so a failure during private recovery needs a node restart before recovery can be retried.
• The log lines for these snapshot loads change: they now read Deserialising snapshot (size: N, public only: true, capturing public maps) and Deserialising snapshot (size: N, public only: false, in place, reusing captured public maps: true|false), so log-based tooling may need updating.
• Extra work: one SHA-256 pass over the public domain when it is installed and another at private recovery, and the size pass when a snapshot is generated, which reads sizes the maps already track.