Add a scratch-backed user data region for host/guest byte exchange

Problem:
Hyperlight embedders that move larger byte payloads across the host/guest boundary currently have to choose between two poor fits. The normal function-call path serializes parameters and returns through FlatBuffers-backed input/output stacks, which is a good contract for structured arguments but adds serialization/deserialization and stack-buffer overhead for large byte arrays. The init-data/user-memory path is also not appropriate as a mutable per-call exchange buffer: init data belongs to snapshot-backed guest state, and after copy-on-write snapshots the live guest-written bytes may no longer be readable by a simple host memcpy from the original shared region.

The missing primitive was an opt-in, bounded, host-addressable scratch buffer whose lifecycle matches transient per-call data: the host can write bytes, the guest can read or mutate them in place, the host can read the result, and restore clears the buffer because scratch is not snapshot state.

What changed:
This change adds a User Data region as a fixed-capacity slab in scratch memory, placed after the existing input and output buffers and before the copied page-table / allocator area. The capacity is configured with `SandboxConfiguration::set_user_data_size`, defaults to zero, participates in scratch minimum-size checks, and is included in layout compatibility so snapshots cannot be restored into a sandbox with a mismatched user-data capacity.

The region is advertised to guests through the PEB as a `GuestMemoryRegion`. Host code gets `MultiUseSandbox::user_data_size`, `write_user_data`, and `read_user_data`. Rust guests get capacity/pointer helpers plus bounded read/write helpers and non-allocating borrowed access helpers for larger payloads. C guests get `hl_user_data_size()` and `hl_user_data_ptr()`, intentionally exposing pointer-and-capacity metadata for caller-bounded raw access.

The implementation preserves the default behavior for existing users: capacity zero means no usable region, no additional public action is required, and the input/output buffer layout is unchanged except for the appended zero-sized user-data descriptor. When a positive capacity is configured, scratch sizing must be large enough to hold input, output, user data, fixed scratch overhead, and copied page tables.

Lifecycle and API contract:
The supported exchange pattern is: host writes bytes with `write_user_data`, guest code reads or mutates the region during a mutating/non-restoring call, then the host reads bytes with `read_user_data` before any restore. Restore clears scratch, so user data is cleared both on explicit restore and through convenience call paths that snapshot/restore around the call.

Host reads and writes start at offset zero and are whole-buffer operations from the caller's perspective. Oversized reads/writes fail before accessing adjacent scratch memory. Successful shorter writes intentionally preserve bytes after `data.len()` rather than clearing the tail. This keeps the region a raw application-managed buffer; callers that use variable-length payloads must track logical length or explicitly clear unused bytes when stale tail bytes would matter.

`MultiUseSandbox::from_snapshot` treats user-data capacity as a layout compatibility field rather than as an ignored runtime override. If the caller supplies a config whose user-data capacity differs from the snapshot layout, construction fails with the existing snapshot-layout mismatch error. This matches restore behavior and avoids surprising reuse of snapshots under incompatible raw-buffer contracts.

Security considerations and justification:
The feature stays within Hyperlight's hypervisor isolation model: the guest still only sees memory already mapped into the VM, and the host APIs bound all host copies by configured capacity. Rust guest helpers also check the PEB-advertised capacity before producing borrowed or owned access. The C API is deliberately lower-level: it exposes a raw pointer plus capacity and documents that C callers must bound their own memory operations.

The important security model change is that user data is a raw byte channel, not a FlatBuffers-verified call/return message. Bytes read by the host from user data must be treated as guest-controlled and validated by the embedder's application protocol before influencing memory access, host function calls, authorization decisions, or other security-sensitive behavior. The security docs and developer guidance now call this out explicitly.

Because short writes preserve the tail, stale bytes can leak between logical requests if an embedder reuses a sandbox and reads beyond the current logical payload length. This is an intentional raw-buffer semantic, but it is documented and tested. Embedders with tenant/request boundaries should restore, clear the full region, or maintain an application-level length field and never expose bytes past it.

The feature does not add VM/page-level guard isolation around user data. That was considered, but it would require separate mappings or page-permission changes rather than reusing the existing scratch slot. For the first release, the chosen security boundary is helper/API-level bounds over scratch memory, matching Hyperlight's existing scratch-buffer model.

Performance comparison:
A sibling sample-workload benchmark was added for the existing 24 KiB input / 8 KiB output workload. The original benchmark passes a 24 KiB byte vector through the FlatBuffers call path and returns an 8 KiB byte vector. The new sibling benchmark writes the 24 KiB input through `write_user_data`, calls a guest function that reads the region in place and reports an 8 KiB logical output, then reads the 8 KiB result with `read_user_data`.

A quick local Criterion run with `cargo bench -p hyperlight-host --bench benchmarks -- sample_workloads --warm-up-time 1 --measurement-time 2 --sample-size 10` showed:

- FlatBuffers C guest: about 89.9 microseconds per call
- FlatBuffers Rust guest: about 90.5 microseconds per call
- User data C guest: about 69.2 microseconds per call
- User data Rust guest: about 72.4 microseconds per call

That run shows the user-data path roughly 20-23 microseconds faster for this sample workload. The exact number is environment-dependent, but the direction matches the design goal: avoid FlatBuffers serialization of large byte payloads while keeping the unavoidable VM call overhead.

Alternatives considered:
1. Keep using init data with CoW-aware host reads. This was rejected because init data is snapshot-backed guest state, not transient per-call I/O. Making host reads follow CoW page-table state would add page walking and CR3/cache complexity, and would continue to blur the semantic boundary between snapshot state and scratch exchange data.

2. Exclude or special-case GuestBlob/init data from snapshots. This was rejected as too invasive: init data lives in the primary snapshot region, and carving out a writable non-snapshotted subregion would affect memory layout, snapshot construction, restore behavior, compatibility checks, and VM mappings for all users.

3. Use writable `map_region` or a separate mapping slot. This was rejected for the first cut because writable mappings are not the existing supported pattern, require separate lifecycle and memory-slot handling, and are heavier than reusing scratch memory that is already host-addressable and guest-visible.

4. Split the feature into two directional raw regions: host-write/guest-read and guest-write/host-read. This was considered in planning and rejected for this change because the primary use case is in-place transformation of a shared byte buffer. The split design gives clearer ownership for request/response protocols but doubles configuration, layout, PEB, helper, compatibility, and test surface area; it also still would not provide VM-enforced directional permissions without a larger mapping redesign.

5. Zero-fill the tail on every successful short host write. This was rejected in favor of raw buffer semantics. Preserving the tail avoids hidden work and lets applications manage subregion layout themselves. The trade-off is documented clearly: callers must track logical length or explicitly clear when stale bytes matter.

6. Provide only an allocating Rust guest read helper. This was rejected for large/hot paths after review. The final Rust guest surface includes non-allocating closure-based accessors so guests can inspect or mutate large configured regions without allocating a full `Vec` in the guest heap; the allocating helper remains as a convenience for smaller reads.

Validation:
The implementation includes configuration and layout tests for zero, one-byte, non-page-aligned 4097-byte, 64 KiB, and 1 MiB capacities. Host memory-manager tests cover fresh-zero reads, exact read/write, oversized read/write rejection, atomic oversized-write failure, and preserved tail behavior after short writes. Integration tests exercise Rust and C guests for capacity discovery, guest-visible zeroes, host-to-guest-to-host mutation, half-capacity mutation, capacity+1 rejection, and failed-call semantics. Restore tests cover explicit restore clearing, deprecated convenience-call restore clearing, capacity mismatch rejection, and `from_snapshot` behavior.

Verification performed during the change included formatting with the repository rustfmt toolchain, debug and release builds, all-target/all-feature clippy, Rust and C guest builds in debug and release, targeted debug and release user-data tests, i686 checks, and a final Society-of-Thought review with follow-up fixes for snapshot compatibility, stale-tail documentation, guest capacity+1 coverage, non-allocating guest helpers, scratch-sizing documentation, and security guidance.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Quetzal Bradley <qbradley@qbradley.com>

Author: qbradley

CHANGELOG.md

@@ -4,6 +4,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Prerelease] - Unreleased
 
+### Added
+* User data region APIs for transient host/guest byte exchange, including host read/write methods and Rust/C guest discovery helpers.
+
 ## [v0.15.0] - 2026-05-06
 
 ### Added

README.md

@@ -53,7 +53,7 @@ fn say_hello(name: String) -> Result<String> {
 }
 ```
 
-To get started, see the [Getting Started](./docs/getting-started.md) guide. For more details on writing guests, see [How to build a Hyperlight guest binary](./docs/how-to-build-a-hyperlight-guest-binary.md).
+To get started, see the [Getting Started](./docs/getting-started.md) guide. For more details on writing guests, see [How to build a Hyperlight guest binary](./docs/how-to-build-a-hyperlight-guest-binary.md). For execution internals, including the user data region, see [How code gets executed in a VM](./docs/hyperlight-execution-details.md).
 
 ## When to use Hyperlight
 

docs/README.md

@@ -26,7 +26,7 @@ This project is composed internally of several components, depicted in the below
 
 * [Getting Started](./getting-started.md)
 * [Glossary](./glossary.md)
-* [How code gets executed in a VM](./hyperlight-execution-details.md)
+* [How code gets executed in a VM](./hyperlight-execution-details.md), including the user data region
 * [How to build a Hyperlight guest binary](./how-to-build-a-hyperlight-guest-binary.md)
 * [Security considerations](./security.md)
 * [Technical requirements document](./technical-requirements-document.md)

docs/hyperlight-execution-details.md

@@ -35,6 +35,18 @@ At the highest level, Hyperlight takes roughly the following steps to create and
    1. In the former case, exit successfully
    2. In any of the latter cases, exit with a failure message
 
+## User data region
+
+Sandboxes can optionally reserve a user data region for transient host/guest byte exchange. The default capacity is zero; embedders configure a positive capacity with `SandboxConfiguration::set_user_data_size`. Fresh regions read as zeroes, and larger capacities may require increasing scratch size to satisfy the sandbox's existing scratch memory limits.
+
+When configured, the region is advertised to the guest through the PEB and can be accessed from the host with `MultiUseSandbox::write_user_data`, `MultiUseSandbox::read_user_data`, and `MultiUseSandbox::user_data_size`. Host reads and writes start at the beginning of the region and fail if the caller's buffer exceeds the configured capacity. Successful shorter writes preserve existing tail bytes, so callers should track logical payload length or clear unused bytes explicitly.
+
+The region is scratch memory, so it is not captured in snapshots. A restore clears the region, including restore performed by convenience call paths that restore the sandbox before returning to the host. Snapshot restore compatibility includes the configured capacity and reports the existing layout-mismatch error for capacity mismatches; `MultiUseSandbox::from_snapshot` rejects a caller-supplied configuration whose user data capacity differs from the snapshot layout. The intended exchange pattern is host write, mutating guest call, then host read before restore; after a failed guest call, callers should treat region contents as application-defined unless their protocol defines a successful handoff.
+
+Rust guest helpers expose bounded access to the region, including non-allocating borrowed access for larger payloads. The C guest API exposes `hl_user_data_size()` and `hl_user_data_ptr()`; C callers must use the reported size to bound copies. These APIs are helper-level bounds, not VM guard-page isolation around the region.
+
+Representative tests cover 4097-byte, 64 KiB, and 1 MiB capacities across Rust and C guests. Larger capacities should be measured for the application's expected copy and restore costs.
+
 ---
 
 _<sup>[1]</sup> nearly universal support_

docs/security-guidance-for-developers.md

@@ -10,6 +10,7 @@ This document discusses the security requirements and best practices for service
 * All host functions that receive parameters from a guest, or operate indirectly on guest data _MUST_ be continuously fuzzed
 * Host functions _MUST NOT_ call APIs or be used to expose functionality deemed risky in a multi-tenant context
 * Guests and host processes _MUST_ use the same version of a FlatBuffer definition
+* Raw user data region bytes _MUST_ be treated as guest-controlled and validated by the host protocol before use
 
 More detailed guidance on the requirements and best practices is detailed below.
 
@@ -56,6 +57,10 @@ We emit this recommendation because there is a history of compiler bugs, which m
 
 For Rust code, if the return code is InvalidFlatBuffer, the input _MUST_ be rejected.
 
+## User data region – raw bytes must be validated before use
+
+The user data region intentionally bypasses the FlatBuffers call/return schema so applications can exchange large raw byte payloads. Hosts that read from the user data region _MUST_ treat those bytes as tainted guest-controlled input. Any length fields, offsets, nested formats, or semantic claims encoded in the region _MUST_ be validated by the host protocol before they are used to access memory, call host functions, or influence security-sensitive decisions. Hosts _SHOULD_ restore the sandbox or clear the full configured region between tenants or requests when stale bytes would be sensitive.
+
 ## Flatbuffers – the host process _MUST NOT_ operate on Flatbuffers from several threads.
 
 Because of the zero-copy approach that FlatBuffers is using, there is a risk of memory safety issues. Flatbuffers are unsafe to be used in a multithreaded environment. This is explicitly indicated in several parts of the Flatbuffer documentation.

docs/security.md

@@ -14,6 +14,8 @@ Hyperlight runs all guest code inside a Virtual Machine, Each VM only has access
 
 All communication between the host and the guest is done through a shared memory buffer. Messages are serialized and deserialized using [FlatBuffers](https://flatbuffers.dev/). To minimize attack surface area, we rely on FlatBuffers to formally specify the data structures passed to/from the host and guest, and to generate serialization/deserialization code. Of course, a compromised guest can write arbitrary data to the shared memory buffer, but the host will not accept anything that does not match our strongly typed FlatBuffer [schemas](../src/schema).
 
+The optional user data region is an explicit exception to the FlatBuffers-mediated call protocol. It exposes raw bytes in shared scratch memory so embedders can define their own payload format. Hosts must treat bytes read from user data as guest-controlled, validate any application-level format before use, and clear or restore the region between security boundaries when stale bytes would be sensitive.
+
 ### Accessing host functionality from the guest
 
 Hyperlight provides a mechanism for the host to register functions that may be called from the guest. This mechanism is useful to allow developers to provide guests with strictly controlled access to functionality we don't make available by default inside the VM. This mechanism likely represents the largest attack surface area of this project.

src/hyperlight_common/src/arch/aarch64/layout.rs

@@ -20,6 +20,10 @@ pub const SNAPSHOT_PT_GVA_MIN: usize = 0xffff_8000_0000_0000;
 pub const SNAPSHOT_PT_GVA_MAX: usize = 0xffff_80ff_ffff_ffff;
 pub const MAX_GPA: usize = 0x0000_000f_ffff_ffff;
 
-pub fn min_scratch_size(_input_data_size: usize, _output_data_size: usize) -> usize {
+pub fn min_scratch_size(
+    _input_data_size: usize,
+    _output_data_size: usize,
+    _user_data_size: usize,
+) -> usize {
     unimplemented!("min_scratch_size")
 }

src/hyperlight_common/src/arch/amd64/layout.rs

@@ -38,7 +38,11 @@ pub const MAX_GPA: usize = 0x0000_000f_ffff_ffff;
 /// - (up to) 3 pages for mapping that
 /// - Two pages for the exception stack and metadata
 /// - A page-aligned amount of memory for I/O buffers (for now)
-pub fn min_scratch_size(input_data_size: usize, output_data_size: usize) -> usize {
-    (input_data_size + output_data_size).next_multiple_of(crate::vmem::PAGE_SIZE)
+pub fn min_scratch_size(
+    input_data_size: usize,
+    output_data_size: usize,
+    user_data_size: usize,
+) -> usize {
+    (input_data_size + output_data_size + user_data_size).next_multiple_of(crate::vmem::PAGE_SIZE)
         + 12 * crate::vmem::PAGE_SIZE
 }

src/hyperlight_common/src/arch/i686/layout.rs

@@ -24,7 +24,11 @@ pub const MAX_GPA: usize = 0xFEDF_FFFF;
 /// Minimum scratch region size: IO buffers (page-aligned) plus 12 pages
 /// for bookkeeping and the exception stack. Page table space is validated
 /// separately by `set_pt_size()`.
-pub fn min_scratch_size(input_data_size: usize, output_data_size: usize) -> usize {
-    (input_data_size + output_data_size).next_multiple_of(crate::vmem::PAGE_SIZE)
+pub fn min_scratch_size(
+    input_data_size: usize,
+    output_data_size: usize,
+    user_data_size: usize,
+) -> usize {
+    (input_data_size + output_data_size + user_data_size).next_multiple_of(crate::vmem::PAGE_SIZE)
         + 12 * crate::vmem::PAGE_SIZE
 }

src/hyperlight_common/src/mem.rs

@@ -79,10 +79,13 @@ pub struct HyperlightPEB {
     /// PEB struct).
     #[cfg(feature = "nanvix-unstable")]
     pub file_mappings: GuestMemoryRegion,
+    pub user_data: GuestMemoryRegion,
 }
 
 #[cfg(test)]
 mod tests {
+    use std::mem::{offset_of, size_of};
+
     use super::*;
 
     #[test]
@@ -109,11 +112,29 @@ mod tests {
                 size: 0x9999,
                 ptr: 0xaaaa,
             },
+            user_data: GuestMemoryRegion {
+                size: 0xbbbb,
+                ptr: 0xcccc,
+            },
         };
         let bytes = bytemuck::bytes_of(&peb);
         let peb2 = *bytemuck::from_bytes::<HyperlightPEB>(bytes);
         let peb2_bytes = bytemuck::bytes_of(&peb2);
         assert_eq!(peb, peb2);
         assert_eq!(bytes, peb2_bytes);
     }
+
+    #[test]
+    fn user_data_is_appended_to_peb() {
+        #[cfg(feature = "nanvix-unstable")]
+        assert_eq!(
+            offset_of!(HyperlightPEB, user_data),
+            size_of::<GuestMemoryRegion>() * 5
+        );
+        #[cfg(not(feature = "nanvix-unstable"))]
+        assert_eq!(
+            offset_of!(HyperlightPEB, user_data),
+            size_of::<GuestMemoryRegion>() * 4
+        );
+    }
 }