Skip to content

docs: update docs for gpu deployments #9483

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rasswanth-s merged 1 commit into from
Aug 2, 2026
Merged

docs: update docs for gpu deployments #9483

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions notebooks/enclave/gemma/colab/1. DO-model-owner-gemma-restrict.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@
"metadata": {},
"outputs": [],
"source": [
"# Debug enclaves (start-debug / tf-apply-dev) run with encryption off — use encryption=False there.\n",
"model_owner = login_do(encryption=True)\n",
"print(f\" Model owner : {model_owner.email}\")"
]
Expand Down
1 change: 1 addition & 0 deletions notebooks/enclave/gemma/colab/2. DO-benchmark-owner-gemma-restrict.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@
"metadata": {},
"outputs": [],
"source": [
"# Debug enclaves (start-debug / tf-apply-dev) run with encryption off — use encryption=False there.\n",
"benchmark_owner = login_do(encryption=True)\n",
"print(f\" Benchmark owner : {benchmark_owner.email}\")"
]
Expand Down
13 changes: 13 additions & 0 deletions packages/syft-enclave/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,19 @@ just start-debug EMAIL my-vm n2d-standard-4 # override name / machine type
just stop [name] # Teardown: Deletes the VM.
```

Debug enclaves run with encryption off — data owner clients must match: `login_do(encryption=False)`.

## GPU deployments

Prefix any deploy with `hardware=gpu` to switch from the CPU default to `a3-highgpu-1g` (1× H100 80GB, Intel TDX):

```bash
just hardware=gpu start EMAIL # production
just hardware=gpu start-debug EMAIL # debug
```

GPU enclaves use flex-start provisioning: the create call may wait for H100 capacity (up to 2h), then the VM runs `gpu_run_duration_seconds` (default 2 days). Details: [docs/terraform.md — GPU deployments](docs/terraform.md#gpu-deployments).

## Inspect a running VM

All inspect commands take an optional `name` (default: `syft-enclave-vm`). Zone is always read from `settings.json`.
Expand Down
3 changes: 2 additions & 1 deletion packages/syft-enclave/docs/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,10 @@ curl http://EXTERNAL_IP:8080/attestation | python3 -m json.tool

The response includes:

- `attestation.hardware.hwmodel` - TEE hardware type (`GCP_AMD_SEV`)
- `attestation.hardware.hwmodel` - TEE hardware type (`GCP_AMD_SEV`; `GCP_INTEL_TDX` on gpu deployments)
- `attestation.hardware.secboot` - Secure boot status
- `attestation.hardware.dbgstat` - Debug status (`enabled` for debug image, `disabled-since-boot` for production)
- `attestation.container.image_digest` - SHA256 of the running container image
- `attestation.nvidia_gpu` - gpu deployments only: `cc_mode` (`"ON"` = confidential computing active), `gpus[].hwmodel` (`GCP_NVIDIA_H100`), driver version
- `attestation.gce.*` - GCP project, zone, instance info
- `raw_token` - Full JWT for independent verification against Google's JWKS
24 changes: 14 additions & 10 deletions packages/syft-enclave/docs/dev.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,15 @@ Debug image features:
The debug image allows SSH access and container log redirection to serial output, making it easier to troubleshoot issues.

```bash
just start-debug
just start-debug # cpu (default)
just hardware=gpu start-debug # gpu (a3-highgpu-1g, 1x H100) — see docs/terraform.md "GPU deployments"
```

Debug enclaves run with encryption off — data owner clients must match: `login_do(encryption=False)`.

## Building & pushing a new Docker Image

The image must be built for `linux/amd64` since GCP Confidential VMs run on AMD EPYC CPUs. On Apple Silicon, use the multi-arch recipe so the same tag works locally (arm64) and on GCP (amd64):
The image must be built for `linux/amd64` — GCP Confidential VMs are amd64 (AMD EPYC for cpu deployments, Intel Sapphire Rapids for gpu). On Apple Silicon, use the multi-arch recipe so the same tag works locally (arm64) and on GCP (amd64):

```bash
just build-push # multi-arch (linux/amd64 + linux/arm64)
Expand All @@ -37,11 +40,12 @@ Both push to `docker.io/openminedreleasebot/syft-client-enclave:latest`. You nee

## Troubleshooting

| Symptom | Cause | Fix |
| ------------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `exec format error` in serial logs | Docker image built for wrong architecture (arm64 on amd64 VM) | Rebuild with `just build-push` (multi-arch) or `just build-push-amd` |
| `unexpected_snp_attestation` | Used `SEV_SNP` instead of `SEV` | Confidential Spaces only supports `SEV` and `TDX`, not `SEV_SNP` |
| `logging redirection only allowed on debug environment` | Used `tee-container-log-redirect=true` with production image | Use `just start-debug` instead of `just start` |
| `403 Forbidden` pulling image | VM service account lacks Artifact Registry access | Grant `roles/artifactregistry.reader` or use a public Docker Hub image |
| VM terminates immediately | Container crashed with `tee-restart-policy=Never` | Switch to `just start-debug` to investigate |
| `OnHostMaintenance` error | Missing `--maintenance-policy` flag | The recipes already set `--maintenance-policy=MIGRATE` for SEV; check you didn't override |
| Symptom | Cause | Fix |
| ------------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| `exec format error` in serial logs | Docker image built for wrong architecture (arm64 on amd64 VM) | Rebuild with `just build-push` (multi-arch) or `just build-push-amd` |
| `unexpected_snp_attestation` | Used `SEV_SNP` instead of `SEV` | Confidential Spaces only supports `SEV` and `TDX`, not `SEV_SNP` |
| `logging redirection only allowed on debug environment` | Used `tee-container-log-redirect=true` with production image | Use `just start-debug` instead of `just start` |
| `403 Forbidden` pulling image | VM service account lacks Artifact Registry access | Grant `roles/artifactregistry.reader` or use a public Docker Hub image |
| VM terminates immediately | Container crashed with `tee-restart-policy=Never` | Switch to `just start-debug` to investigate |
| `OnHostMaintenance` error | Missing `--maintenance-policy` flag | The recipes already set `--maintenance-policy=TERMINATE`; check you didn't override |
| GPU VM suddenly gone | Flex-start run duration elapsed (VM + disk auto-delete) | By design — redeploy with `just hardware=gpu start-debug` |
12 changes: 6 additions & 6 deletions packages/syft-enclave/docs/enclave_architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This directory contains a Docker image that packages `syft-client` with an HTTP

```
┌─────────────────────────────────────────────────────┐
│ GCP Confidential VM (AMD SEV - encrypted memory) │
│ GCP Confidential VM (SEV/TDX - encrypted memory) │
│ │
│ ┌───────────────────────────────────────────────┐ │
│ │ Confidential Space OS (hardened, read-only) │ │
Expand All @@ -33,9 +33,9 @@ This directory contains a Docker image that packages `syft-client` with an HTTP

Confidential Spaces supports the following confidential compute types:

| Type | Description | Machine Types |
| ----- | ----------------------------------- | ------------------- |
| `SEV` | AMD Secure Encrypted Virtualization | `n2d-*` (AMD Milan) |
| `TDX` | Intel Trust Domain Extensions | `c3-*` |
| Type | Description | Machine Types |
| ----- | ----------------------------------- | ------------------------------------- |
| `SEV` | AMD Secure Encrypted Virtualization | `n2d-*` (AMD Milan) |
| `TDX` | Intel Trust Domain Extensions | `c3-*`; `a3-highgpu-1g` (1× H100 GPU) |

> **Note:** AMD SEV-SNP is NOT supported by Confidential Spaces (only by raw Confidential VMs). The recipes use `SEV`.
> **Note:** AMD SEV-SNP is NOT supported by Confidential Spaces (only by raw Confidential VMs). The recipes use `SEV` for cpu deployments and `TDX` for gpu (`hardware=gpu`).
38 changes: 30 additions & 8 deletions packages/syft-enclave/docs/terraform.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ A declarative alternative to the gcloud-based Justfile recipes (`init` / `provis
- Required GCP APIs (Compute, Confidential Computing, Secret Manager, IAM)
- The enclave service account and its IAM roles
- The Secret Manager secret **including the token version**
- The Confidential Space VM (AMD SEV)
- The Confidential Space VM (AMD SEV for `cpu`, Intel TDX for `gpu` — see [GPU deployments](#gpu-deployments))

State is **local** (`terraform/terraform.tfstate`, gitignored): one operator = one state file = one enclave deployment. The `.tf` configuration is shared via git; your per-deployment values live in a gitignored `terraform.tfvars`. The gcloud recipes remain available — but don't manage the _same_ resources with both flows.

Expand Down Expand Up @@ -43,12 +43,28 @@ Then fill in:
| `data_owners` | List of data-owner emails; **all** must approve every job. |
| `token_file` | Absolute path to the enclave's Google Drive token JSON (see [auth.md](../../../docs/auth.md) to create one). |

Optional overrides (commented in the example file): `vm_name`, `machine_type`, `boot_disk_size_gb`, `image_repo`, `image_tag`, `image_digest`, `use_encryption`, `job_timeout_seconds`. The deployed image is `image_repo:image_tag` (default `:latest`); when `image_digest` is set it takes precedence and the image is pinned as `image_repo@sha256:...`.
Optional overrides (commented in the example file): `hardware`, `max_run_duration_seconds`, `vm_name`, `machine_type`, `boot_disk_size_gb`, `image_repo`, `image_tag`, `image_digest`, `use_encryption`, `job_timeout_seconds`. The deployed image is `image_repo:image_tag` (default `:latest`); when `image_digest` is set it takes precedence and the image is pinned as `image_repo@sha256:...`.

Do **not** set `dev_mode` in tfvars — pass it on the command line (`-var=dev_mode=false` for production, `just tf-apply-dev` for dev), which takes precedence, so a stray tfvars value can never produce the wrong deployment type.

> ⚠️ **Token handling.** The token content is uploaded to Secret Manager _and_ stored in plaintext in the local `terraform.tfstate`. The state file is gitignored — never commit it, share it, or move it off your machine.

## GPU deployments

Set `hardware = "gpu"` in `terraform.tfvars` — **in tfvars, never via `-var`**: a later plain `tf-apply` without the flag would silently replace the VM back to CPU. This switches to the only GPU configuration Confidential Space supports:

| | `cpu` (default) | `gpu` |
| ------------ | -------------------------- | ------------------------------------------ |
| Machine | `n2d-standard-2` (AMD SEV) | `a3-highgpu-1g` — 1× H100 80GB (Intel TDX) |
| Provisioning | on-demand | flex-start (queued, no on-demand exists) |

Flex-start semantics: the apply queues for H100 capacity (it may wait; there is no fallback), then the VM runs uninterrupted for `max_run_duration_seconds` (default 2 days, max 7), after which **the VM and disk are auto-deleted** — redeploy to continue. Jobs must fit that window (the default job timeout of 30 days exceeds it).

Prerequisites:

- Quota (both default to 0 — request first): regional `PREEMPTIBLE_NVIDIA_H100_GPUS` + global `GPUS_ALL_REGIONS`.
- A supported zone (as of Aug 2026): `us-central1-a`, `us-east5-a`, `europe-west4-c`.

## Quickstart: production

Hardened image — no SSH, TEE enforcement, encryption on, container restart policy `Never`.
Expand Down Expand Up @@ -85,6 +101,8 @@ Deletes the VM, secret, service account, and IAM bindings. The enabled APIs are

Debug image — SSH enabled, container logs redirected to serial output, encryption off (override with `use_encryption = true` in tfvars), restart policy `Always`.

> Clients must match the enclave's encryption setting: against a debug enclave, data owners log in with `login_do(encryption=False)`.

```bash
just tf-apply-dev
just tf-logs # full container logs via SSH + journalctl (falls back to serial)
Expand Down Expand Up @@ -116,12 +134,16 @@ just tf-attest # attestation report, fetched via SSH + localhost

## Troubleshooting

| Error | Fix |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `could not find default credentials` | Run `gcloud auth application-default login` — `gcloud auth login` alone is not enough. |
| `oauth2: "invalid_grant"` | ADC token expired/revoked — re-run `gcloud auth application-default login`. |
| Quota project warnings | `gcloud auth application-default set-quota-project YOUR_PROJECT_ID` |
| Attestation fails on first boot | Check the enclave SA has `roles/confidentialcomputing.workloadUser` (Terraform grants it to the dedicated SA only, not the default compute SA). A fresh apply waits 120s for IAM propagation (`time_sleep.iam_propagation`); if the launcher still 403s and exits (`exit_code=4`, dead VM), reset the VM: `gcloud compute instances reset <vm_name>`. |
| Error | Fix |
| ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `could not find default credentials` | Run `gcloud auth application-default login` — `gcloud auth login` alone is not enough. |
| `oauth2: "invalid_grant"` | ADC token expired/revoked — re-run `gcloud auth application-default login`. |
| Quota project warnings | `gcloud auth application-default set-quota-project YOUR_PROJECT_ID` |
| Attestation fails on first boot | Check the enclave SA has `roles/confidentialcomputing.workloadUser` (Terraform grants it to the dedicated SA only, not the default compute SA). A fresh apply waits 120s for IAM propagation (`time_sleep.iam_propagation`); if the launcher still 403s and exits (`exit_code=4`, dead VM), reset the VM: `gcloud compute instances reset <vm_name>`. |
| GPU: quota exceeded on apply | Request regional `PREEMPTIBLE_NVIDIA_H100_GPUS` + global `GPUS_ALL_REGIONS` quota (both default to 0). |
| GPU: `GPU Driver installation is not supported` in launcher logs | Known Confidential Space issue — restart the VM. |
| GPU: attestation `mismatched measurement record at index 9` | Known issue — full **stop/start** of the VM (a guest reboot is not enough). |
| GPU: VM/state suddenly gone | Flex-start `max_run_duration_seconds` elapsed — the VM and disk auto-delete by design. Redeploy. |

## Formatting and validation

Expand Down
Loading