From bb140d3be49c1d7c626d31a25b05e5b0e7c55308 Mon Sep 17 00:00:00 2001 From: rasswanth-s <43314053+rasswanth-s@users.noreply.github.com> Date: Sun, 2 Aug 2026 18:53:20 +0530 Subject: [PATCH] update docs for gpu deployments --- .../1. DO-model-owner-gemma-restrict.ipynb | 1 + ...2. DO-benchmark-owner-gemma-restrict.ipynb | 1 + packages/syft-enclave/README.md | 13 +++++++ packages/syft-enclave/docs/api.md | 3 +- packages/syft-enclave/docs/dev.md | 24 +++++++----- .../syft-enclave/docs/enclave_architecture.md | 12 +++--- packages/syft-enclave/docs/terraform.md | 38 +++++++++++++++---- 7 files changed, 67 insertions(+), 25 deletions(-) diff --git a/notebooks/enclave/gemma/colab/1. DO-model-owner-gemma-restrict.ipynb b/notebooks/enclave/gemma/colab/1. DO-model-owner-gemma-restrict.ipynb index 405387a475b..51568c49755 100644 --- a/notebooks/enclave/gemma/colab/1. DO-model-owner-gemma-restrict.ipynb +++ b/notebooks/enclave/gemma/colab/1. DO-model-owner-gemma-restrict.ipynb @@ -108,6 +108,7 @@ "metadata": {}, "outputs": [], "source": [ + "# Debug enclaves (start-debug / tf-apply-dev) run with encryption off — use encryption=False there.\n", "model_owner = login_do(encryption=True)\n", "print(f\" Model owner : {model_owner.email}\")" ] diff --git a/notebooks/enclave/gemma/colab/2. DO-benchmark-owner-gemma-restrict.ipynb b/notebooks/enclave/gemma/colab/2. DO-benchmark-owner-gemma-restrict.ipynb index 6ae3ce14dd1..55f38287a67 100644 --- a/notebooks/enclave/gemma/colab/2. DO-benchmark-owner-gemma-restrict.ipynb +++ b/notebooks/enclave/gemma/colab/2. DO-benchmark-owner-gemma-restrict.ipynb @@ -89,6 +89,7 @@ "metadata": {}, "outputs": [], "source": [ + "# Debug enclaves (start-debug / tf-apply-dev) run with encryption off — use encryption=False there.\n", "benchmark_owner = login_do(encryption=True)\n", "print(f\" Benchmark owner : {benchmark_owner.email}\")" ] diff --git a/packages/syft-enclave/README.md b/packages/syft-enclave/README.md index 3d83348afb9..94e03062a09 100644 --- a/packages/syft-enclave/README.md +++ b/packages/syft-enclave/README.md @@ -58,6 +58,19 @@ just start-debug EMAIL my-vm n2d-standard-4 # override name / machine type just stop [name] # Teardown: Deletes the VM. ``` +Debug enclaves run with encryption off — data owner clients must match: `login_do(encryption=False)`. + +## GPU deployments + +Prefix any deploy with `hardware=gpu` to switch from the CPU default to `a3-highgpu-1g` (1× H100 80GB, Intel TDX): + +```bash +just hardware=gpu start EMAIL # production +just hardware=gpu start-debug EMAIL # debug +``` + +GPU enclaves use flex-start provisioning: the create call may wait for H100 capacity (up to 2h), then the VM runs `gpu_run_duration_seconds` (default 2 days). Details: [docs/terraform.md — GPU deployments](docs/terraform.md#gpu-deployments). + ## Inspect a running VM All inspect commands take an optional `name` (default: `syft-enclave-vm`). Zone is always read from `settings.json`. diff --git a/packages/syft-enclave/docs/api.md b/packages/syft-enclave/docs/api.md index 56a9c183b08..c7aaa41482b 100644 --- a/packages/syft-enclave/docs/api.md +++ b/packages/syft-enclave/docs/api.md @@ -41,9 +41,10 @@ curl http://EXTERNAL_IP:8080/attestation | python3 -m json.tool The response includes: -- `attestation.hardware.hwmodel` - TEE hardware type (`GCP_AMD_SEV`) +- `attestation.hardware.hwmodel` - TEE hardware type (`GCP_AMD_SEV`; `GCP_INTEL_TDX` on gpu deployments) - `attestation.hardware.secboot` - Secure boot status - `attestation.hardware.dbgstat` - Debug status (`enabled` for debug image, `disabled-since-boot` for production) - `attestation.container.image_digest` - SHA256 of the running container image +- `attestation.nvidia_gpu` - gpu deployments only: `cc_mode` (`"ON"` = confidential computing active), `gpus[].hwmodel` (`GCP_NVIDIA_H100`), driver version - `attestation.gce.*` - GCP project, zone, instance info - `raw_token` - Full JWT for independent verification against Google's JWKS diff --git a/packages/syft-enclave/docs/dev.md b/packages/syft-enclave/docs/dev.md index 03125836886..18c271189c9 100644 --- a/packages/syft-enclave/docs/dev.md +++ b/packages/syft-enclave/docs/dev.md @@ -21,12 +21,15 @@ Debug image features: The debug image allows SSH access and container log redirection to serial output, making it easier to troubleshoot issues. ```bash -just start-debug +just start-debug # cpu (default) +just hardware=gpu start-debug # gpu (a3-highgpu-1g, 1x H100) — see docs/terraform.md "GPU deployments" ``` +Debug enclaves run with encryption off — data owner clients must match: `login_do(encryption=False)`. + ## Building & pushing a new Docker Image -The image must be built for `linux/amd64` since GCP Confidential VMs run on AMD EPYC CPUs. On Apple Silicon, use the multi-arch recipe so the same tag works locally (arm64) and on GCP (amd64): +The image must be built for `linux/amd64` — GCP Confidential VMs are amd64 (AMD EPYC for cpu deployments, Intel Sapphire Rapids for gpu). On Apple Silicon, use the multi-arch recipe so the same tag works locally (arm64) and on GCP (amd64): ```bash just build-push # multi-arch (linux/amd64 + linux/arm64) @@ -37,11 +40,12 @@ Both push to `docker.io/openminedreleasebot/syft-client-enclave:latest`. You nee ## Troubleshooting -| Symptom | Cause | Fix | -| ------------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | -| `exec format error` in serial logs | Docker image built for wrong architecture (arm64 on amd64 VM) | Rebuild with `just build-push` (multi-arch) or `just build-push-amd` | -| `unexpected_snp_attestation` | Used `SEV_SNP` instead of `SEV` | Confidential Spaces only supports `SEV` and `TDX`, not `SEV_SNP` | -| `logging redirection only allowed on debug environment` | Used `tee-container-log-redirect=true` with production image | Use `just start-debug` instead of `just start` | -| `403 Forbidden` pulling image | VM service account lacks Artifact Registry access | Grant `roles/artifactregistry.reader` or use a public Docker Hub image | -| VM terminates immediately | Container crashed with `tee-restart-policy=Never` | Switch to `just start-debug` to investigate | -| `OnHostMaintenance` error | Missing `--maintenance-policy` flag | The recipes already set `--maintenance-policy=MIGRATE` for SEV; check you didn't override | +| Symptom | Cause | Fix | +| ------------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------------------------- | +| `exec format error` in serial logs | Docker image built for wrong architecture (arm64 on amd64 VM) | Rebuild with `just build-push` (multi-arch) or `just build-push-amd` | +| `unexpected_snp_attestation` | Used `SEV_SNP` instead of `SEV` | Confidential Spaces only supports `SEV` and `TDX`, not `SEV_SNP` | +| `logging redirection only allowed on debug environment` | Used `tee-container-log-redirect=true` with production image | Use `just start-debug` instead of `just start` | +| `403 Forbidden` pulling image | VM service account lacks Artifact Registry access | Grant `roles/artifactregistry.reader` or use a public Docker Hub image | +| VM terminates immediately | Container crashed with `tee-restart-policy=Never` | Switch to `just start-debug` to investigate | +| `OnHostMaintenance` error | Missing `--maintenance-policy` flag | The recipes already set `--maintenance-policy=TERMINATE`; check you didn't override | +| GPU VM suddenly gone | Flex-start run duration elapsed (VM + disk auto-delete) | By design — redeploy with `just hardware=gpu start-debug` | diff --git a/packages/syft-enclave/docs/enclave_architecture.md b/packages/syft-enclave/docs/enclave_architecture.md index cbdf55e267e..6f3a22f0903 100644 --- a/packages/syft-enclave/docs/enclave_architecture.md +++ b/packages/syft-enclave/docs/enclave_architecture.md @@ -6,7 +6,7 @@ This directory contains a Docker image that packages `syft-client` with an HTTP ``` ┌─────────────────────────────────────────────────────┐ -│ GCP Confidential VM (AMD SEV - encrypted memory) │ +│ GCP Confidential VM (SEV/TDX - encrypted memory) │ │ │ │ ┌───────────────────────────────────────────────┐ │ │ │ Confidential Space OS (hardened, read-only) │ │ @@ -33,9 +33,9 @@ This directory contains a Docker image that packages `syft-client` with an HTTP Confidential Spaces supports the following confidential compute types: -| Type | Description | Machine Types | -| ----- | ----------------------------------- | ------------------- | -| `SEV` | AMD Secure Encrypted Virtualization | `n2d-*` (AMD Milan) | -| `TDX` | Intel Trust Domain Extensions | `c3-*` | +| Type | Description | Machine Types | +| ----- | ----------------------------------- | ------------------------------------- | +| `SEV` | AMD Secure Encrypted Virtualization | `n2d-*` (AMD Milan) | +| `TDX` | Intel Trust Domain Extensions | `c3-*`; `a3-highgpu-1g` (1× H100 GPU) | -> **Note:** AMD SEV-SNP is NOT supported by Confidential Spaces (only by raw Confidential VMs). The recipes use `SEV`. +> **Note:** AMD SEV-SNP is NOT supported by Confidential Spaces (only by raw Confidential VMs). The recipes use `SEV` for cpu deployments and `TDX` for gpu (`hardware=gpu`). diff --git a/packages/syft-enclave/docs/terraform.md b/packages/syft-enclave/docs/terraform.md index 979b8d692ec..d1e0cdf5349 100644 --- a/packages/syft-enclave/docs/terraform.md +++ b/packages/syft-enclave/docs/terraform.md @@ -5,7 +5,7 @@ A declarative alternative to the gcloud-based Justfile recipes (`init` / `provis - Required GCP APIs (Compute, Confidential Computing, Secret Manager, IAM) - The enclave service account and its IAM roles - The Secret Manager secret **including the token version** -- The Confidential Space VM (AMD SEV) +- The Confidential Space VM (AMD SEV for `cpu`, Intel TDX for `gpu` — see [GPU deployments](#gpu-deployments)) State is **local** (`terraform/terraform.tfstate`, gitignored): one operator = one state file = one enclave deployment. The `.tf` configuration is shared via git; your per-deployment values live in a gitignored `terraform.tfvars`. The gcloud recipes remain available — but don't manage the _same_ resources with both flows. @@ -43,12 +43,28 @@ Then fill in: | `data_owners` | List of data-owner emails; **all** must approve every job. | | `token_file` | Absolute path to the enclave's Google Drive token JSON (see [auth.md](../../../docs/auth.md) to create one). | -Optional overrides (commented in the example file): `vm_name`, `machine_type`, `boot_disk_size_gb`, `image_repo`, `image_tag`, `image_digest`, `use_encryption`, `job_timeout_seconds`. The deployed image is `image_repo:image_tag` (default `:latest`); when `image_digest` is set it takes precedence and the image is pinned as `image_repo@sha256:...`. +Optional overrides (commented in the example file): `hardware`, `max_run_duration_seconds`, `vm_name`, `machine_type`, `boot_disk_size_gb`, `image_repo`, `image_tag`, `image_digest`, `use_encryption`, `job_timeout_seconds`. The deployed image is `image_repo:image_tag` (default `:latest`); when `image_digest` is set it takes precedence and the image is pinned as `image_repo@sha256:...`. Do **not** set `dev_mode` in tfvars — pass it on the command line (`-var=dev_mode=false` for production, `just tf-apply-dev` for dev), which takes precedence, so a stray tfvars value can never produce the wrong deployment type. > ⚠️ **Token handling.** The token content is uploaded to Secret Manager _and_ stored in plaintext in the local `terraform.tfstate`. The state file is gitignored — never commit it, share it, or move it off your machine. +## GPU deployments + +Set `hardware = "gpu"` in `terraform.tfvars` — **in tfvars, never via `-var`**: a later plain `tf-apply` without the flag would silently replace the VM back to CPU. This switches to the only GPU configuration Confidential Space supports: + +| | `cpu` (default) | `gpu` | +| ------------ | -------------------------- | ------------------------------------------ | +| Machine | `n2d-standard-2` (AMD SEV) | `a3-highgpu-1g` — 1× H100 80GB (Intel TDX) | +| Provisioning | on-demand | flex-start (queued, no on-demand exists) | + +Flex-start semantics: the apply queues for H100 capacity (it may wait; there is no fallback), then the VM runs uninterrupted for `max_run_duration_seconds` (default 2 days, max 7), after which **the VM and disk are auto-deleted** — redeploy to continue. Jobs must fit that window (the default job timeout of 30 days exceeds it). + +Prerequisites: + +- Quota (both default to 0 — request first): regional `PREEMPTIBLE_NVIDIA_H100_GPUS` + global `GPUS_ALL_REGIONS`. +- A supported zone (as of Aug 2026): `us-central1-a`, `us-east5-a`, `europe-west4-c`. + ## Quickstart: production Hardened image — no SSH, TEE enforcement, encryption on, container restart policy `Never`. @@ -85,6 +101,8 @@ Deletes the VM, secret, service account, and IAM bindings. The enabled APIs are Debug image — SSH enabled, container logs redirected to serial output, encryption off (override with `use_encryption = true` in tfvars), restart policy `Always`. +> Clients must match the enclave's encryption setting: against a debug enclave, data owners log in with `login_do(encryption=False)`. + ```bash just tf-apply-dev just tf-logs # full container logs via SSH + journalctl (falls back to serial) @@ -116,12 +134,16 @@ just tf-attest # attestation report, fetched via SSH + localhost ## Troubleshooting -| Error | Fix | -| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `could not find default credentials` | Run `gcloud auth application-default login` — `gcloud auth login` alone is not enough. | -| `oauth2: "invalid_grant"` | ADC token expired/revoked — re-run `gcloud auth application-default login`. | -| Quota project warnings | `gcloud auth application-default set-quota-project YOUR_PROJECT_ID` | -| Attestation fails on first boot | Check the enclave SA has `roles/confidentialcomputing.workloadUser` (Terraform grants it to the dedicated SA only, not the default compute SA). A fresh apply waits 120s for IAM propagation (`time_sleep.iam_propagation`); if the launcher still 403s and exits (`exit_code=4`, dead VM), reset the VM: `gcloud compute instances reset `. | +| Error | Fix | +| ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `could not find default credentials` | Run `gcloud auth application-default login` — `gcloud auth login` alone is not enough. | +| `oauth2: "invalid_grant"` | ADC token expired/revoked — re-run `gcloud auth application-default login`. | +| Quota project warnings | `gcloud auth application-default set-quota-project YOUR_PROJECT_ID` | +| Attestation fails on first boot | Check the enclave SA has `roles/confidentialcomputing.workloadUser` (Terraform grants it to the dedicated SA only, not the default compute SA). A fresh apply waits 120s for IAM propagation (`time_sleep.iam_propagation`); if the launcher still 403s and exits (`exit_code=4`, dead VM), reset the VM: `gcloud compute instances reset `. | +| GPU: quota exceeded on apply | Request regional `PREEMPTIBLE_NVIDIA_H100_GPUS` + global `GPUS_ALL_REGIONS` quota (both default to 0). | +| GPU: `GPU Driver installation is not supported` in launcher logs | Known Confidential Space issue — restart the VM. | +| GPU: attestation `mismatched measurement record at index 9` | Known issue — full **stop/start** of the VM (a guest reboot is not enough). | +| GPU: VM/state suddenly gone | Flex-start `max_run_duration_seconds` elapsed — the VM and disk auto-delete by design. Redeploy. | ## Formatting and validation