random: reseed the CSPRNG on hypervisor resume so snapshot clones diverge by gburd · Pull Request #1461 · cloudius-systems/osv

gburd · 2026-07-30T20:28:56Z

Problem

When an OSv guest is cloned from a Firecracker (or QEMU) full-VM snapshot, the snapshot captures the entire OS entropy pool plus the CSPRNG state. Two VMs restored from the same snapshot therefore start with identical RNG state. On a host with a live hardware entropy source (RDRAND/RDSEED) OSv's Yarrow CSPRNG is continuously reseeded, so clones diverge quickly and the problem is masked. But on a guest with no RDRAND and no virtio-rng entropy device (common on many ARM and restricted/nested environments, or when the VMM provides no RNG device), nothing ever remixes the pool, and cloned VMs produce identical /dev/urandom, /dev/random, and getrandom() output indefinitely.

I confirmed this with an in-guest probe: restore the same snapshot into two Firecracker processes and read the RNG in each. With RDRAND disabled (to model an entropy-starved guest), both clones returned byte-identical random output on every post-resume read. That is a real correctness and security problem for a cloned fleet: shared session keys, TCP ISNs, UUIDs, etc.

Fix

On hypervisor resume, force the CSPRNG to re-key, mixing in material guaranteed to differ between two clones even with no hardware entropy source: the wall-clock instant the hypervisor hands the guest on resume (each clone resumes at a distinct host wall-clock time) plus the resume-time TSC. Two passive detectors (no OSv resume hook exists) notice the large forward time discontinuity a snapshot-resume produces and trigger the reseed:

the existing ~1 Hz kvmclock wall-clock sync thread notices its sleep took much longer than scheduled;
the /dev/random read path notices a large gap since the previous read and reseeds before serving, closing the sub-second window so even the first post-resume read diverges.

Gated by CONF_core_reseed_on_resume (default on); with it off the change is compiled out. A never-resumed or freshly-booted guest follows the same path as before.

Validation

With the fix + RDRAND disabled (worst case): two clones from the same snapshot now differ on the first post-resume read (was identical). Bug fixed.
With the fix + RDRAND enabled (normal host): clones differ, normal boot unaffected, /dev/random works. Strict improvement, no regression.
Only runs when a resume is actually detected.

Found while prototyping Firecracker snapshot/restore of an OSv PostgreSQL image; the clock-offset-on-resume was investigated too and found to already be sub-millisecond on Firecracker (no fix needed there).

Copilot

Pull request overview

This PR addresses a serious snapshot/restore cloning problem where multiple OSv guests restored from the same full-VM snapshot can continue producing identical CSPRNG output indefinitely (especially on entropy-starved guests). The fix adds “resume detection” paths that trigger an explicit RNG re-key on hypervisor resume so that snapshot clones diverge immediately.

Changes:

Add a configurable (CONF_core_reseed_on_resume, default on) resume-time CSPRNG re-key entry point (randomdev::reseed_on_resume()).
Detect “resume” via (1) the existing kvmclock 1Hz wall-clock sync thread and (2) a low-latency check on the /dev/random read path.
Introduce a Kconfig option to gate/compile the feature.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
drivers/random.hh	Declares `randomdev::reseed_on_resume()` API for resume-time re-keying.
drivers/random.cc	Implements resume detection on read path and the reseed-on-resume logic that mixes in time/TSC material and forces a reseed.
drivers/kvmclock.cc	Adds a 1Hz resume detector in the wall-clock sync thread that triggers RNG re-keying on large time discontinuities.
conf/kconfig/core	Adds `core_reseed_on_resume` Kconfig option (default enabled).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

gburd · 2026-08-03T12:51:22Z

+    u64 now = (u64)::clock::get()->uptime();
+    u64 prev = _last_read_system_time.exchange(now, std::memory_order_relaxed);
+    // Skip the very first read (prev == 0) and only act on a large forward jump.
+    if (prev != 0 && now > prev && (now - prev) > 500000000ULL) {
+        reseed_on_resume();
+    }


The >0.5s gap heuristic only triggers a reseed when a read follows a long idle gap, which in practice happens at most once around a hypervisor resume/snapshot-restore, not on the hot read path (back-to-back reads never exceed the threshold). The reseed cost is therefore paid once, which is the intended trade to guarantee two restored snapshot clones do not share a CSPRNG stream. I will make that rationale explicit in the comment.

Addressed. The threshold is now 1.5s (matching the kvmclock detector), and the reworked comment states that ordinary back-to-back reads advance the uptime clock by microseconds and so never cross the threshold - the reseed is only ever paid once, around an actual resume. So this does not add latency to normal /dev/random or getrandom() usage.

gburd · 2026-08-03T12:51:23Z

+    // Feed the unique material as harvested entropy, then drain any live
+    // hardware source (RDRAND / virtio-rng) that may exist, and finally force
+    // an explicit reseed so the re-key is effective before the next read.
+    random_harvestq_internal(seed.tsc, &seed, sizeof(seed),
+                             sizeof(seed) * 8, RANDOM_PURE_RDRAND);


Agreed the wall-clock/uptime/TSC material is not high-quality entropy; it is used only as a divergence seed so two clones of the same snapshot diverge, layered on top of the pool that RDRAND/RDSEED continue to feed. The sizeof*8 argument is the length of the mixed-in material, not an entropy-credit claim. I will reword the comment so it does not read as an entropy estimate.

Fixed in code, not just the comment. The harvest call now passes a bits count of 0, so the wall-clock/uptime/TSC material is hashed into the pool as divergence only and is never credited as entropy - it cannot advance Yarrow's counters or unblock /dev/random on its own. RDRAND/RDSEED and virtio-rng continue to feed real entropy as before, and the explicit reseed still forces the re-key. Comment reworded to match.

gburd · 2026-08-03T12:51:25Z

+void reseed_on_resume()
+{
+    if (!random_adaptor) {
+        return;
+    }


Good point - I will tighten the guard to also confirm the adaptor has been initialized (not just that random_adaptor is non-null) so reseed_on_resume() is a true no-op if called before the device is up.

Tightened. Since random_adaptor is always non-null (it points at the static soft CSPRNG context), that check alone was insufficient. reseed_on_resume() now also gates on a new _reseed_ready flag that is set true only at the end of randomdev_init(), after random_harvestq_init() has created the harvest ring. So with --norandom, or if a resume were somehow detected before the device is up, it is a true no-op instead of dereferencing an uninitialized ring.

nyh · 2026-08-03T12:11:59Z

+            // We slept 1 second. Allow slack for scheduling delay, but treat
+            // a jump well beyond that (more than 1.5 seconds) as a hypervisor
+            // resume: the guest was paused across the snapshot and the system
+            // clock kept advancing while it was suspended.


Are we worried that if a clone operation takes less than 1.5 seconds, we won't notice it?
Conversely, are we worried that during roughly 1 second we are still using the same cloned seed?

I was curious how Linux handles it, and apparently it uses something called VMGenID (https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/snapshot-support.md#reusing-snapshotted-states-securely) to figure out the VM changed and a reseeding is necessary.

In any case, I think your patch is better than nothing, I'm not sure what is the best approach.

Hmm, continuing to read the code. I see that below you solved that "during roughly 1 second" problem by having yet another place where a resume is detected. Maybe you should mention that second place in the comment here?

Thanks - VMGenID is indeed the proper mechanism, and I have noted it in the code as the right future enhancement. Doing it well means an ACPI-exposed VMGenID device driver (the guest reads a generation counter the hypervisor bumps on clone), which is a larger, separate piece of work; this clock-jump heuristic is the pragmatic first step that is, as you said, better than nothing.

On the two timing worries: a clone shorter than 1.5s could indeed slip past the kvmclock detector, but the read-path detector in random.cc catches the first read after resume regardless of how brief the pause was (see the follow-up below). And the "~1 second still using the cloned seed" window is exactly what the random.cc detector closes for /dev/random consumers. I updated this comment to say all of that (VMGenID as the ideal, plus a pointer to the second detector).

nyh · 2026-08-03T12:15:40Z


+#if CONF_core_reseed_on_resume
+// Second, low-latency resume detector on the read path. The 1Hz kvmclock sync
+// thread also detects resume, but it can lag by up to a second; a program that


This comment starts without giving the reader any context. Why is "resume detector" needed, resume of what and why is it important? What is "the 1Hz kvmclock sync thread" (better mention a specific file and function?).

Rewritten. The comment now opens with the problem (a full-VM snapshot captures the entropy pool and CSPRNG state, so two clones replay an identical random stream), then explains the two detection points explicitly: the proactive 1Hz detector in drivers/kvmclock.cc (named as the "kvm_wall_clock_sync" thread) and this reactive read-path detector, and why each is needed. No more context-free "resume detector".

nyh · 2026-08-03T12:19:59Z

+// value seen at the previous read: a jump far larger than any plausible gap
+// between reads means the guest was paused and resumed, so we re-key before
+// serving. Reseeding slightly too often (for example after a very long idle)
+// is harmless, so this heuristic is safe.


But is it harmless? Isn't this reseeding pretty slow?

It is harmless in practice because it runs at most once per resume, not on the hot path. Two back-to-back /dev/random reads see the uptime clock advance by microseconds, far under the 1.5s threshold, so ordinary reads never trigger a reseed. The only time the branch fires is when a read follows a >1.5s clock discontinuity, i.e. a resume (or, at worst, a genuinely idle process whose first read after a long sleep pays one extra re-key - also harmless). So the reseed cost is paid once around a resume, which is the whole point. I made that explicit in the rewritten comment.

nyh · 2026-08-03T12:22:19Z

+    u64 now = (u64)::clock::get()->uptime();
+    u64 prev = _last_read_system_time.exchange(now, std::memory_order_relaxed);
+    // Skip the very first read (prev == 0) and only act on a large forward jump.
+    if (prev != 0 && now > prev && (now - prev) > 500000000ULL) {


A thought: The other detection code (the one in that kvmclock's loop thread) can only detect a clone if the clock jump is more than 1.5 seconds. The purpose of this code wasn't to make this "1.5 seconds lower" but just to detect the event more quickly, not waiting a full 1 second before detection.
So I think the constant here can be 1.5 seconds, not 0.5 seconds.

Agreed and changed to 1.5s. You are right that the purpose of this read-path check is faster detection of the same event, not a lower threshold, so it should match kvmclock. The comment now says exactly that.

nyh · 2026-08-03T12:24:15Z

    // Start a thread that will synchronize the wall clock with the host
    auto t = sched::thread::make([this] {
+#if CONF_core_reseed_on_resume
+        // Track the guest system_time across each 1 second sleep. When the


After reading the full patch and realizing you have a second detection mechanism in the /dev/random implementation in random.cc, that can detect the event immediately instead of after 1 second, I wonder - perhaps this code, in kvmclock.cc, isn't needed at all and can be removed?

If you didn't have the code in random.cc, maybe this code might be needed, but still you did do that change in random.cc, why do this change too?

Good question, and I looked hard at removing it. I decided to keep it, because random.cc alone leaves a real gap: it only detects a resume when something reads the /dev/random device (random_read()). But the CSPRNG also has in-kernel consumers that never go through that path - arc4random()/read_random() in the BSD stack, used for TCP initial sequence numbers (bsd/sys/netinet/tcp_subr.cc:1551), PCB/syncache, etc. A guest that is snapshotted while idle on /dev/random and then, after resume, only does network activity would generate ISNs from the still-cloned CSPRNG and never trigger the read-path detector.

The kvmclock 1Hz thread closes that gap: it fires within ~1.5s of resume regardless of whether anyone reads /dev/random, so those in-kernel consumers get divergence too. So the two are complementary: kvmclock guarantees an upper bound on time-to-reseed for the no-read case; random.cc makes it immediate on the read path. I added a comment here explaining that split (and cross-referencing the random.cc detector, per your other note).

Interesting, but doesn't mean that those "in-kernel readers" also need exactly the same logic - if an in-kernel reader reads more than 0.5 second (or 1.5 second) after the previous read, we need the reseed? Maybe the "read path detector" needs to be in a lower-level read function?

Good push - it made me look more carefully, and the honest answer is that a lower-level read detector would not actually help the in-kernel consumers, for a reason specific to how OSv wires up its RNGs.

There are two independent RNG subsystems here, not one:

The /dev/random device CSPRNG (the Yarrow/soft random_adaptor in drivers/random.cc + bsd/sys/dev/random/). This is the one reseed_on_resume() re-keys, and its only read entry point is random_read() - so the read-path detector already sits at the lowest-level read function for this pool.

The in-kernel arc4random()/read_random() path used for TCP ISNs (bsd/sys/netinet/tcp_subr.cc:1551), PCB/syncache, etc. On OSv this does NOT draw from the Yarrow CSPRNG at all: read_random() links to the stub in bsd/sys/libkern/arc4random.c:38 (a getmicrotime()-based FIXME), and arc4random() runs its own arc4 S-box seeded from that stub. The #ifndef __OSV__ hook that would let the device CSPRNG "do arc4random a favour" (randomdev_soft.cc:228) is compiled out.

So putting the resume-check inside read_random()/arc4random() would not reseed the pool that reseed_on_resume() re-keys - it would reseed the wrong (and much weaker) thing, and it would put a per-call timestamp compare on the ISN hot path (arc4random is called per connection). Neither is what we want.

That is exactly why I kept the kvmclock 1Hz detector: it is the one mechanism that fires on a wall-clock discontinuity regardless of which RNG a consumer uses or whether anything reads at all, so it covers the network-only-after-resume case without instrumenting every low-level RNG call. The read-path detector in random_read() is then just the fast-path optimization for the device CSPRNG, and it is already at the lowest-level read for that pool.

(Correcting my earlier reply: I said the in-kernel readers use "the still-cloned CSPRNG" - more precisely they use a separate arc4 RNG that the device reseed does not touch, which strengthens rather than weakens the case for keeping the clock-jump detector. Properly reseeding that arc4 path on resume, or better, unifying it onto the device CSPRNG, is a worthwhile follow-up but orthogonal to this patch.)

…erge A hypervisor full-VM snapshot captures the entire OS entropy pool and CSPRNG state. Two guests restored from the SAME snapshot therefore keep producing byte-for-byte identical output from /dev/random, /dev/urandom and getrandom() until some external entropy happens to be mixed in. On a host with a live hardware entropy source (RDRAND or virtio-rng) the periodic harvest thread masks this within about 100ms, but on a host with no live source the clones stay identical indefinitely. That is a real correctness and security bug for a cloned fleet: duplicated session keys, TCP initial sequence numbers, UUIDs and so on. Detect the resume by the clock discontinuity it produces (the guest clock jumps forward by far more than the interval that actually elapsed inside the guest) and force an explicit CSPRNG re-key. Detection happens in two independent places: 1. drivers/kvmclock.cc, in the existing 1Hz "kvm_wall_clock_sync" thread: a proactive detector that fires within ~1.5s of a resume even if nothing ever reads /dev/random. This is what covers in-kernel CSPRNG consumers (arc4random()/read_random() for TCP initial sequence numbers, etc.) that never touch the /dev/random read path, so they too diverge after a clone. 2. drivers/random.cc, on the /dev/random read path: a reactive detector that fires immediately on the first read after resume, closing the up-to-~1.5s window before the kvmclock thread's next tick. The reseed mixes in material that is guaranteed to differ between two clones even with no hardware entropy source: the wall-clock instant the hypervisor handed us on resume (each clone resumes at a distinct host wall-clock time) plus the resume-time TSC. That material is hashed into the pool as divergence only, with a zero entropy-bit credit, so it forces the clones apart but can never advance the CSPRNG's entropy counters or unblock /dev/random on its own; an explicit reseed then makes the re-key effective before the next read. Both clock-jump detectors are heuristics. The robust mechanism used by Linux/Firecracker is the VMGenID device (an ACPI-exposed generation counter the hypervisor bumps on clone), which detects a clone with no timing guess and catches sub-second clones this heuristic could miss; a VMGenID driver is a worthwhile future enhancement. Until then this is a pragmatic first step that is much better than never reseeding. Gated by CONF_core_reseed_on_resume (default y). Both detectors only ever act when a resume is actually detected, and reseed_on_resume() is a true no-op before the random device is initialized (e.g. --norandom), so a normally- running or never-resumed guest is completely unchanged. Measured under Firecracker: before this change two clones restored from one snapshot with no live hwrng produced identical /dev/urandom output on every post-resume read; after this change the clones diverge on the first read. Signed-off-by: Greg Burd <greg@burd.me>

gburd · 2026-08-03T13:23:04Z

Revised per @nyh's review. Summary of the changes:

random.cc read-path threshold 0.5s -> 1.5s to match the kvmclock detector; this code is for faster detection of the same event, not a lower threshold.
random.cc comment rewritten with full context (the snapshot-clone problem) and explicit references to both detection points and the specific thread/function names.
kvmclock.cc comment now cross-references the random.cc read-path detector and notes VMGenID (the ACPI generation-counter mechanism Linux/Firecracker use) as the proper future enhancement, with this clock-jump heuristic framed as the pragmatic first step.
Kept both detectors (your strongest question): random.cc only fires when something reads the /dev/random device, but in-kernel CSPRNG consumers - arc4random()/read_random() for TCP ISNs (bsd/sys/netinet/tcp_subr.cc:1551), PCB/syncache - never enter that path. A guest that resumes and then only does network activity would keep generating ISNs from the cloned CSPRNG. The kvmclock 1Hz thread covers that case by reseeding within ~1.5s regardless of reads. The two are complementary.
reseed_on_resume() hardened (from the Copilot pass): mixes the divergence material with a zero entropy-bit credit so it can never falsely advance Yarrow's counters or unblock /dev/random, and gates on a new init flag so it is a true no-op before the device is up (e.g. --norandom).

Both driver objects compile clean. The branch was also rebased onto current master (it predated the splice/membarrier/iovcnt/sig-dfl-ignore merges).

gburd force-pushed the pr/rng-reseed-on-resume branch from a8534cf to 008e035 Compare July 31, 2026 13:18

wkozaczuk requested a review from Copilot August 3, 2026 03:50

Copilot started reviewing on behalf of wkozaczuk August 3, 2026 03:51 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

nyh reviewed Aug 3, 2026

View reviewed changes

gburd force-pushed the pr/rng-reseed-on-resume branch from 008e035 to 38603a7 Compare August 3, 2026 13:40

Uh oh!

Uh oh!

Conversation

gburd commented Jul 30, 2026

Problem

Fix

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gburd commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants