diff --git a/README.md b/README.md index 0acd4b4..346b0ec 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # NVIDIA CCluster Docs -This repository contains the [Fern](https://buildwithfern.com/learn/docs/getting-started/overview) source for the NVIDIA CCluster documentation site, published at `docs.nvidia.com/ccluster`. +This repository contains the [Fern](https://buildwithfern.com/learn/docs/getting-started/overview) source for the NVIDIA CCluster documentation site, published at [docs.nvidia.com/ccluster](https://docs.nvidia.com/ccluster). The site uses the NVIDIA global theme (`global-theme: nvidia`), so logo, colors, fonts, layout, and footer come from the shared [global-theme](https://gitlab-master.nvidia.com/fern-documentation/global-theme) repository rather than this repo. See the [NVIDIA Fern onboarding guide](https://gitlab-master.nvidia.com/fern-documentation/guides) for platform details. @@ -52,7 +52,7 @@ python3 tools/check-docs-paths --allow '../docs/' docs fern 2. Add new pages to the `navigation` section of [fern/docs.yml](fern/docs.yml) with a `../docs/
/.mdx` path; only pages listed there are published. 3. Put images in the section's `images/` folder and reference them with `./images/...` paths. Absolute paths and `..` segments are rejected by `tools/check-docs-paths`; the only exception is the `../docs/` prefix of page paths in `fern/docs.yml`. 4. Use Fern's [built-in components](https://buildwithfern.com/learn/docs/writing-content/components/overview) (`Card`, `CardGroup`, `Frame`, `Note`, `Tip`, and others). Icons are Font Awesome classes such as `fa-solid fa-lock`. -5. Link between pages with root-relative paths such as `/apps/llm`; Fern adds the `/ccluster` base path. +5. Link between pages with root-relative paths such as `/apps/dynamo`; Fern adds the `/ccluster` base path. ## Release Process @@ -64,4 +64,4 @@ Publishing runs through GitHub Actions and needs the `FERN_TOKEN` repository sec | [preview-docs.yml](.github/workflows/preview-docs.yml) | Same-repo PRs | Posts a Fern preview link on the PR | | [publish-docs.yml](.github/workflows/publish-docs.yml) | Merge to `main` that changes `docs/` or `fern/`, or manual run | Publishes production after the checks pass | -Connecting the Fern site to `docs.nvidia.com/ccluster` requires a review and Jira ticket from the Developer Experience team; ask in [#cdd-fern](https://nvidia.enterprise.slack.com/archives/C0A8987CQ7K). +Merged changes go live at [docs.nvidia.com/ccluster](https://docs.nvidia.com/ccluster), the custom domain configured in [fern/docs.yml](fern/docs.yml). For Fern platform or domain issues, ask in [#cdd-fern](https://nvidia.enterprise.slack.com/archives/C0A8987CQ7K). diff --git a/docs/apps/compute.mdx b/docs/apps/compute.mdx index bea7d85..2062437 100644 --- a/docs/apps/compute.mdx +++ b/docs/apps/compute.mdx @@ -17,9 +17,9 @@ Your image does not need an SSH server preinstalled. CCluster injects one at dep ### Private registry images -To deploy an image from a private registry (e.g., Amazon ECR, Google Artifact Registry, Azure Container Registry, or a self-hosted registry), provide credentials in the **Image Registry Username** and **Image Registry Password** fields. CCluster automatically detects the registry from the image URL. For public images, leave both fields empty. +To deploy an image from a private registry (e.g., Amazon ECR, Google Artifact Registry, Azure Container Registry, or a self-hosted registry), set **Image pull authentication**. Choose **Username and password** and select **Image Registry Credentials** saved in [Secrets](/resources/vault), or choose **Existing cluster pull secret** and enter the **Cluster pull secret name** that your cluster administrator manages. CCluster automatically detects the registry from the image URL. For public images, leave authentication unset. -API deployments can instead [reference an existing image pull secret](/resources/custom-image#use-an-existing-image-pull-secret) managed by your cluster administrator using `image_pull_secret_name`. +API deployments can also [reference an existing image pull secret](/resources/custom-image#use-an-existing-image-pull-secret) using `image_pull_secret_name`. ## 2. Configure access and deploy @@ -27,11 +27,12 @@ Enter your SSH public key to configure access to the instance, select a GPU inst Optional settings on the create form: +- **Storage** — mount filesystem volumes or S3 buckets on the **Hardware** step. Mount a writable volume at `/home` to keep your home files when the instance is paused. See [Managing Storage](/resources/storage). - **Enable logging** — collect container output and show it on the deployment **Logs** tab. - **Annotations** — optional key/value metadata on the instance. - Compute instance launch page + Compute Instance Hardware step with cluster, hardware cards, and storage ## 3. SSH into the instance @@ -68,7 +69,7 @@ scp root@:/home/results.csv . ssh -L 6006:localhost:6006 root@ ``` -X11 forwarding and SSH agent forwarding are not supported. +SSH agent forwarding is supported. X11 forwarding is not supported. The interactive shell depends on what your image provides: @@ -84,7 +85,7 @@ The instance includes the libraries from your image. Additional packages can be While the instance is **Active**, the deployment **Terminal** tab opens a shell in the browser without SSH from your machine. Use SSH (above) for `scp`/`SFTP` and port forwarding. See [Managing Deployments](/resources/managing-deployments) for Terminal availability, pause/resume, clone, and logs. -Compute deployments are not revision-aware in the console: edits update the instance in place and there is no rollback table. +Compute deployments are revision-aware: spec edits create a new revision, and the **Detail** tab lists revisions so you can inspect them and roll back. See [Managing Deployments](/resources/managing-deployments#revisions-rollback-and-rollout). ## What's next @@ -98,17 +99,17 @@ Compute deployments are not revision-aware in the console: edits update the inst In-console Terminal, logs, pause, and clone. Store SSH keys and registry credentials for reuse across deployments - Explore dedicated, protected endpoints for production model deployments. + Serve LLMs with disaggregated prefill/decode, autoscaling, and scale to zero. diff --git a/docs/apps/dynamo.mdx b/docs/apps/dynamo.mdx index a9616ec..06d9719 100644 --- a/docs/apps/dynamo.mdx +++ b/docs/apps/dynamo.mdx @@ -32,7 +32,7 @@ From **New Deployment**, choose **Dynamo**. The **Endpoint** step contains the r - **Endpoint Name** — a unique name containing lowercase letters, numbers, and hyphens. It must begin with a lowercase letter, end with a letter or number, and be no more than 20 characters. - **Model** — the Hugging Face model identifier loaded by the vLLM workers. - **Served Model Name** — an optional alias clients can send in the OpenAI `model` field. If you leave it empty, use the Hugging Face model identifier. -- **Hugging Face Token** — required for gated and private models. +- **Hugging Face Token** — required for gated and private models. Select a token saved in [Secrets](/resources/vault). - **Protocol** — `HTTP` for the OpenAI-compatible API or `gRPC` for the KServe v2 inference protocol. - **Private Endpoint** — select or create a Bearer token or client certificate. - **Annotations** — optional key/value metadata applied to the deployment components. @@ -119,7 +119,7 @@ Prefill and Decode pools do not autoscale, and they cannot scale to zero. ### Storage volumes -Both topologies can mount reusable volumes from the cluster in the **Storage** section of the **Hardware** step. Mounts apply to the Frontend and to every worker's main container, at the same paths. Volumes must belong to the deployment's creator. When you edit a deployment, removing a mount changes only the new revision; the volume and its data are kept. +Both topologies can mount reusable volumes from the cluster in the **Storage** section of the **Hardware** step. To create volumes, see [Managing Storage](/resources/storage). Mounts apply to the Frontend and to every worker's main container, at the same paths. Volumes must belong to the deployment's creator. When you edit a deployment, removing a mount changes only the new revision; the volume and its data are kept. Click **Deploy**. Model download and worker startup can take several minutes, especially for large models that are not already cached. @@ -157,6 +157,10 @@ You can also create a disaggregated deployment through the API. Send a `POST` re Each pool needs a `hardware_instance_id` and a fixed size: set `min_replicas` and `max_replicas` to the same value, at least `1`. The older per-pool `replicas` field is deprecated but still accepted. The Prefill and Decode pools can use the same GPU hardware profile or different profiles, but both profiles must be available in the selected cluster. Every replica consumes one complete hardware instance, so the deployment's GPU footprint is the sum of the Prefill and Decode pool footprints. +To pin the model to a Hugging Face branch, tag, or commit, set `model_revision`. NVIDIA CCluster resolves it to a commit and runs every worker at that commit. + +On clusters that support GPU Direct RDMA, `"weight_transfer": {}` lets a starting worker copy the model weights GPU to GPU from a running worker of the same pool instead of downloading the checkpoint. It pins the model to a single commit, and it needs the same EFA-enabled AWS clusters as `require_gpu_direct_rdma`. When you enable it, `--revision` and `--load-format` in extra arguments are rejected, and environment variable names starting with `MX_` or `MODEL_EXPRESS_`, and `VLLM_PLUGINS`, are reserved. + The `endpoint_bearer_token` in this request protects the deployed inference endpoint. It is separate from the credential used to authenticate the request to the NVIDIA CCluster API. `communications.kv_transfer.policy` selects the KV transfer policy: `allow_tcp`, the console default, or `require_gpu_direct_rdma`. If you omit `communications`, NVIDIA CCluster uses its default NIXL configuration, which also allows TCP. To pin a runtime version, set `runtime_version` to a Dynamo runtime version tag; otherwise NVIDIA CCluster selects the platform default for the cluster and policy. @@ -244,7 +248,7 @@ curl "https:///v1/chat/completions" \ For a certificate-protected endpoint, configure your HTTP client with the downloaded client certificate instead of the `Authorization` header. See [Securing Endpoints](/resources/private). -If you selected `gRPC`, connect to `:443` with TLS and use Dynamo's KServe v2 `ModelInfer` or `ModelStreamInfer` API. This is different from the direct vLLM gRPC API used by General Inference deployments. See the [NVIDIA Dynamo KServe gRPC frontend guide](https://docs.nvidia.com/dynamo/latest/components/frontend/frontend-guide) for the protocol and [Securing Endpoints](/resources/private) for authentication. +If you selected `gRPC`, connect to `:443` with TLS and use Dynamo's KServe v2 `ModelInfer` or `ModelStreamInfer` API. This is different from the direct vLLM gRPC API used by General Inference deployments. See the [NVIDIA Dynamo KServe gRPC frontend guide](https://docs.nvidia.com/dynamo/components/frontend) for the protocol and [Securing Endpoints](/resources/private) for authentication. ## Current scope diff --git a/docs/apps/images/comp-1.png b/docs/apps/images/comp-1.png index a66e909..ffa6208 100644 Binary files a/docs/apps/images/comp-1.png and b/docs/apps/images/comp-1.png differ diff --git a/docs/apps/images/dynamo-advanced-configuration.png b/docs/apps/images/dynamo-advanced-configuration.png index d909029..6fddbb0 100644 Binary files a/docs/apps/images/dynamo-advanced-configuration.png and b/docs/apps/images/dynamo-advanced-configuration.png differ diff --git a/docs/apps/images/dynamo-endpoint-configuration.png b/docs/apps/images/dynamo-endpoint-configuration.png index 80122e6..2f8183e 100644 Binary files a/docs/apps/images/dynamo-endpoint-configuration.png and b/docs/apps/images/dynamo-endpoint-configuration.png differ diff --git a/docs/apps/images/dynamo-hardware-scaling.png b/docs/apps/images/dynamo-hardware-scaling.png index 678065e..a1f6c05 100644 Binary files a/docs/apps/images/dynamo-hardware-scaling.png and b/docs/apps/images/dynamo-hardware-scaling.png differ diff --git a/docs/apps/images/inf-1.png b/docs/apps/images/inf-1.png index f6ded87..94a000a 100644 Binary files a/docs/apps/images/inf-1.png and b/docs/apps/images/inf-1.png differ diff --git a/docs/apps/images/inf-2.png b/docs/apps/images/inf-2.png index c5f29e7..84ccd7e 100644 Binary files a/docs/apps/images/inf-2.png and b/docs/apps/images/inf-2.png differ diff --git a/docs/apps/images/inf-3.png b/docs/apps/images/inf-3.png index 8a9b800..c92c82b 100644 Binary files a/docs/apps/images/inf-3.png and b/docs/apps/images/inf-3.png differ diff --git a/docs/apps/images/inf-4.png b/docs/apps/images/inf-4.png index 3ceb56e..4120bb0 100644 Binary files a/docs/apps/images/inf-4.png and b/docs/apps/images/inf-4.png differ diff --git a/docs/apps/images/llm-stage1.png b/docs/apps/images/llm-stage1.png deleted file mode 100644 index 1201281..0000000 Binary files a/docs/apps/images/llm-stage1.png and /dev/null differ diff --git a/docs/apps/images/llm-stage2.png b/docs/apps/images/llm-stage2.png deleted file mode 100644 index 5f12267..0000000 Binary files a/docs/apps/images/llm-stage2.png and /dev/null differ diff --git a/docs/apps/inference.mdx b/docs/apps/inference.mdx index 09cdc24..7958e9c 100644 --- a/docs/apps/inference.mdx +++ b/docs/apps/inference.mdx @@ -7,7 +7,7 @@ We make it easy for you to containerize and deploy any custom models on CCluster ## 1. Configure your inference deployment -Use the **Add or Select Image** dropdown to choose a pre-configured recipe or manually enter a custom image URL. Similarly, use the **Container Tag** dropdown to pick an available tag or type one in directly. If you select a recipe, all fields except **Add command** are auto-filled (some may be empty). You will still need to provide the entrypoint command under **Optional Details → Add command**. +Use the **Add or Select Image** dropdown to choose a pre-configured recipe or manually enter a custom image URL. Similarly, use the **Container Tag** dropdown to pick an available tag or type one in directly. If you select a recipe, all fields except **Add command** are auto-filled (some may be empty). You will still need to provide the entrypoint command in the **Advanced** step under **Add command**. At a minimum, you need to specify: @@ -15,20 +15,20 @@ At a minimum, you need to specify: - **Container Tag** — the image tag (select from dropdown or enter manually). - **Container Port** — the port your container exposes its HTTP or gRPC service on. - **Protocol** — select `HTTP` (default) or `gRPC`. -- **Health check path** — the endpoint used to verify readiness (e.g., `/health`, `/`). For gRPC deployments, CCluster uses TCP socket checks automatically, so this field is ignored. -- **Image Registry Username / Password** — credentials for private images (e.g., Docker Hub private repo). API deployments can instead [reference an existing image pull secret](/resources/custom-image#use-an-existing-image-pull-secret). +- **Image pull authentication** — for private images, choose **Username and password** and select **Image Registry Credentials** saved in [Secrets](/resources/vault), or choose **Existing cluster pull secret** and enter the **Cluster pull secret name** that your cluster administrator manages. Leave it unset for public images. See [Deploying Custom Models](/resources/custom-image#use-an-existing-image-pull-secret). +- **Hugging Face Token** — select a Hugging Face token saved in [Secrets](/resources/vault) for gated or private models. Do not pass `HF_TOKEN` as a plain environment variable. -The create wizard splits optional settings across **Advanced** and **Scaling**. +The create wizard has three steps: **Endpoint**, **Advanced**, and **Hardware**. Optional runtime settings are in **Advanced**, and **Scaling** settings are in **Hardware**. **Advanced** - **Enable logging** — collect container stdout/stderr and show it on the deployment **Logs** tab. If this is off, that revision has no logs in the console. - **Enable Model Caching** — store model files in a node-local persistent cache. Restarts and replicas scheduled on the same node can reuse the files instead of downloading them again. - **Session affinity** — best-effort sticky routing using the `X-Session-Id` request header. Requests with the same header value land on the same pod, which helps KV-cache reuse for agentic workloads. Requests without the header are routed at random. Affinity is **not** durable: scaling, rollouts, restarts, or readiness-probe transitions can remap sessions to different pods. Do not use it for irreplaceable in-pod state. -- **Health check path** — the endpoint used to verify readiness (also listed in the required fields above). +- **Health check path** — the endpoint used to verify readiness (e.g., `/health`, `/`). For gRPC deployments, CCluster uses TCP socket checks automatically, so this field is ignored. - **Add command** — entrypoint command and arguments. If left empty, the image's default entrypoint is used. - **Metrics port** / **Metrics path** — optional Prometheus scrape endpoint. Set **both** together, or leave both empty to use the defaults (the container port and `/metrics`). -- **Environment variables** — pass additional variables to the container (e.g., `HF_TOKEN`). +- **Environment variables** — pass additional variables to the container. - **Config file** — mount a file into the container. Provide **Filename**, **Mount Path (directory)**, and **File Content**. For worked examples, see [gRPC Inference Endpoints](/resources/grpc) and [Deploying TensorRT-LLM](/resources/trtllm). - **Annotations** — optional key/value metadata on the deployment. @@ -40,7 +40,7 @@ CCluster scales replicas from in-flight request concurrency (**Max Concurrency** | --- | --- | | **Min Replicas** | Lower bound for replica count. `0` allows scale-to-zero when demand drops. | | **Max Replicas** | Upper bound for replica count. Set equal to **Min Replicas** for a fixed size. | -| **Max Concurrency** | Target in-flight requests per replica. Required when **Max Replicas** is greater than **Min Replicas**. | +| **Max Concurrency** | Target in-flight requests per replica. Optional; leave unset for no concurrency target. | | **Cooldown Period (s)** | Scale-down stabilization time, from `0` to `3600`. | @@ -48,11 +48,11 @@ Changing min or max replica counts on a running deployment scales in place and * - General Inference deployment page + General Inference Endpoint step with image, port, protocol, image pull authentication, Hugging Face token, and private endpoint fields -Every deployment must be protected — there are no public endpoints. Secure your endpoint with a Bearer token (an `Authorization: Bearer ` header), an mTLS client certificate, or both. To enable mTLS, select the "Make it a private endpoint?" option, which generates a TLS certificate you can download as a .pem file after the deployment is complete. For more details, see [Securing Endpoints](/resources/private). +Every deployment must be protected — there are no public endpoints. Secure your endpoint with a Bearer token (an `Authorization: Bearer ` header), an mTLS client certificate, or both. To enable mTLS, select the **Private Endpoint** option, which generates a TLS certificate you can download as a .pem file after the deployment is complete. For more details, see [Securing Endpoints](/resources/private). @@ -112,10 +112,10 @@ For the full NIM configuration and an end-to-end walkthrough, see [Deploying NVI By default, NVIDIA CCluster provides several managed clusters and GPU instances for you to deploy your inference containers. - Hardware instance selection for general inference + General Inference Hardware step with cluster, hardware cards, storage, and scaling settings -Select the regional cluster and hardware instance that best fits your need and click Deploy. +Select the regional cluster and hardware instance that best fits your need. To mount persistent storage, add a mount in the **Storage** section below the hardware cards; see [Managing Storage](/resources/storage). Then click Deploy. You can integrate your own private cluster into CCluster through bring-your-own-infrastructure support. To get started, open a support request through your NVIDIA CCluster support channel. @@ -126,7 +126,7 @@ You can integrate your own private cluster into CCluster through bring-your-own- Once deployed, you can see all your deployments under the listing view along with their current status. - Container image configuration for general inference + Deployments list showing a general inference deployment and its status Click on the deployment to view the details page, logs and monitoring information. @@ -183,10 +183,10 @@ For gRPC deployments, see [gRPC Inference Endpoints](/resources/grpc) for connec Learn how to protect your endpoints with Bearer tokens and mTLS certificates - Explore dedicated, protected endpoints for production model deployments. + Serve LLMs with disaggregated prefill/decode, autoscaling, and scale to zero. diff --git a/docs/apps/jobs.mdx b/docs/apps/jobs.mdx index 6447426..88518f4 100644 --- a/docs/apps/jobs.mdx +++ b/docs/apps/jobs.mdx @@ -12,11 +12,11 @@ Under the **Job Information** section, provide the core details for your workloa - **Job Name** — a unique name for the job. It must start with a lowercase letter, contain only lowercase letters, digits, or hyphens (`-`), end with an alphanumeric character, and be at most 20 characters. - **Container Image URL** — the container image to run (e.g., `nvcr.io/nvidia/base/ubuntu`). - **Container Tag** — the image tag (e.g., `latest`). -- **Image Pull Secret Username / Password** — credentials for private images. Leave both empty for public images; if you provide one, you must provide the other. API deployments can instead [reference an existing image pull secret](/resources/custom-image#use-an-existing-image-pull-secret) using `image_pull_secret_name`. +- **Image pull authentication** — for private images, choose **Username and password** and select **Image Registry Credentials** saved in [Secrets](/resources/vault), or choose **Existing cluster pull secret** and enter the **Cluster pull secret name** that your cluster administrator manages. Leave it unset for public images. API deployments can also [reference an existing image pull secret](/resources/custom-image#use-an-existing-image-pull-secret) using `image_pull_secret_name`. ## 2. Set optional details -Expand the **Optional Details** section to tune how the job executes. All of these fields are optional and fall back to sensible defaults. +In the **Advanced** step, tune how the job executes. All of these fields are optional and fall back to sensible defaults. - **Completions** — the number of pods that must complete successfully for the job to be considered done. Defaults to `1`. Use a value greater than `1` for indexed, multi-pod batch work. - **Parallelism** — the maximum number of pods that run at the same time. Defaults to `1` and must be **less than or equal to Completions**. @@ -24,7 +24,7 @@ Expand the **Optional Details** section to tune how the job executes. All of the - **Active Deadline Seconds** — the maximum time (in seconds) the job may run before it is terminated. Must be `1` or greater. Leave empty for no deadline. - **Command** — the entrypoint command and arguments for the container (e.g., `/bin/sh -c python3 "train.py"`). If left empty, the image's default entrypoint is used. - **Enable logging** — toggle to collect and view logs from the job's pods. -- **Environment variables** — pass additional environment variables to the container (e.g., `HF_TOKEN`). +- **Environment variables** — pass additional environment variables to the container. Select saved values from [Secrets](/resources/vault) where you can. - **Annotations** — attach custom key/value metadata to the job. - **Config file** — mount a file into the container. Provide **Filename**, **Mount Path (directory)**, and **File Content**. Use this for startup scripts or small configs your command can read. @@ -41,7 +41,7 @@ For building and deploying your own custom container image on NVIDIA CCluster, s ## 3. Select the cluster and hardware -Choose the regional cluster and GPU hardware instance that best fits your workload, then click **Deploy**. NVIDIA CCluster provides several managed clusters and GPU instance types to run your job on. +Choose the regional cluster and GPU hardware instance that best fits your workload, then click **Deploy**. NVIDIA CCluster provides several managed clusters and GPU instance types to run your job on. To share data between the pods of a job or keep results after the job ends, add a mount in the **Storage** section; the mount applies to every pod of the job. See [Managing Storage](/resources/storage). You can integrate your own private cluster into CCluster through bring-your-own-infrastructure support. To get started, open a support request through your NVIDIA CCluster support channel. @@ -49,7 +49,11 @@ You can integrate your own private cluster into CCluster through bring-your-own- ## 4. Monitor your job -Once created, your job appears in the deployments listing view alongside its current status. Click into the job to view its details page, including per-pod status and, when **Enable logging** is turned on, the logs from each pod. While pods are running, the **Terminal** tab opens an in-browser shell. Jobs are not revision-aware and have no pause/resume or replica autoscaling. See [Managing Deployments](/resources/managing-deployments). +Once created, your job appears in the deployments listing view alongside its current status. Click into the job to view its details page, including per-pod status and, when **Enable logging** is turned on, the logs from each pod. While pods are running, the **Terminal** tab opens an in-browser shell. Jobs can be paused and resumed from the details page, but they are not revision-aware and have no replica autoscaling. See [Managing Deployments](/resources/managing-deployments). + + +Pausing a job deletes its pods, and resuming starts the job again from the beginning. Completed work and in-memory progress are not preserved, so work can run again. To keep progress across a pause, checkpoint in your application and write the checkpoints to persistent storage. See [Managing Storage](/resources/storage). + Because jobs are batch workloads, there is no endpoint to call — the job runs until your container exits. A job is considered complete once the requested number of **Completions** finish successfully, and failed pods are retried up to the **Backoff Limit** (or until the **Active Deadline Seconds** is reached). diff --git a/docs/apps/llm.mdx b/docs/apps/llm.mdx deleted file mode 100644 index e942595..0000000 --- a/docs/apps/llm.mdx +++ /dev/null @@ -1,122 +0,0 @@ ---- -title: 'LLM Serving' -description: 'Plan, optimize, and deploy LLMs effortlessly' ---- - -Deploy dedicated LLM endpoints that fits your performance requirements and budget in just three steps. - -## 1. Choose your LLM -Select or enter the Hugging Face model name of your choosing and provide your Hugging Face token. Also provide a name for the dedicated endpoint you are going to deploy. - - - LLM Serving page showing available models - - - -Make sure you have been granted access to the model you selected. If not, please go to https://huggingface.co/ and request for access. - - -## 2. Plan and optimize -Choose the cluster or the region you want to deploy the model. Based on that, NVIDIA CCluster presents three pre-configured deployment configurations to suit different requirements: - - - LLM Serving deployment configuration options - - -- **Best performance:** A configuration optimized for latency and throughput, suitable for high-demand applications where performance is critical. -- **Lowest cost:** A cost-effective configuration designed to minimize expenses, ideal for non-critical applications with lighter usage. -- **Best value:** A balanced configuration offering a mix of performance and cost efficiency, tailored to provide an ideal trade-off for general usage. - -Each configuration is accompanied by detailed analysis on: - -- **Time to first token:** Indicates the latency between sending a request to a language model and receiving the first piece of its response. -- **Throughput:** Measures the number of requests the model can handle per second. -- **Cost per token:** Shows the cost of generating a million tokens. - -These insights help you choose the configuration that best meets your needs. - -### Optional details - -Expand **Optional Details** for runtime, observability, and scaling controls. These are independent of the preset (or Custom) performance configuration. - -- **Enable logging** — collect container output and show it on the deployment **Logs** tab. -- **Enable Model Caching** — store model files in a node-local persistent cache so restarts and replicas on the same node can skip re-downloading weights. -- **Enable session affinity** — best-effort sticky routing via the `X-Session-Id` request header. Requests with the same header value land on the same pod; requests without it are routed at random. Affinity is not durable across scaling, rollouts, or restarts. -- **Environment variables** — extra variables passed to the serving container. -- **Annotations** — optional key/value metadata. - -| Setting | Description | -| --- | --- | -| **Min Replicas** | Lower bound for replica count. | -| **Max Replicas** | Upper bound for replica count. Set equal to **Min Replicas** for a fixed size. | -| **Max Concurrency** | Target in-flight requests per replica. Required when max is greater than min. | -| **Cooldown Period (s)** | Scale-down stabilization time, from `0` to `3600`. | - - -Scaling min or max replicas on a running deployment does **not** create a new revision. Changing the serving spec does. See [Managing Deployments](/resources/managing-deployments). - - -### (Optional) Performance customization -For advanced users, NVIDIA CCluster also offers an option to customize their model performance configuration. Simply click the "Custom" configuration to gain full control over several tunable parameters. - -## 3. Deploy and integrate -Finally, click "Deploy". Once the deployment is ready in a few minutes, copy the endpoint url and go to `https:///docs` to find the list of API endpoints to start using your LLM deployment. We offer API compatibility with CServe, OpenAI, and Cortex, making integration with other applications seamless. - -From the deployment detail page you can stream logs, open the in-browser **Terminal**, pause or resume, clone, edit (new revision), or roll back. See [Managing Deployments](/resources/managing-deployments). - -Endpoints are never public, so every request must authenticate with a Bearer token or a client certificate. The example below uses a Bearer token; see [Securing Endpoints](/resources/private) for all options. - -```bash showLineNumbers={false} -curl -X 'POST' 'https:///openai/v1/chat/completions' \ - -H 'Authorization: Bearer ' \ - -H 'accept: application/json' \ - -H 'Content-Type: application/json' \ - -d '{ - "messages": [ - { - "role": "user", - "content": "what is the meaning of life?" - } - ], - "model": "meta-llama/Llama-3.2-3B-Instruct", - "max_tokens": 512, - "n": 1, - "presence_penalty": 0, - "stream": true, - "stream_options": { - "include_usage": true - }, - "temperature": 0.7, - "top_p": 1 -}' -``` - -For more details on how to use the LLM deployment, please refer to the [examples](https://github.com/CentML/codex/tree/main/chat-apps) we've prepared. - - - -## What's next - - - - Logs, Terminal, pause, clone, revisions, and replica scaling. - - -Learn how to protect your endpoints with Bearer tokens and mTLS certificates - - - Learn how agents can interact with NVIDIA CCluster services. - - diff --git a/docs/clients/sdk.mdx b/docs/clients/sdk.mdx index 1f3ffea..bd90960 100644 --- a/docs/clients/sdk.mdx +++ b/docs/clients/sdk.mdx @@ -31,11 +31,11 @@ The SDK can also list, create, and delete the **hardware instances** available t - Learn how to generate and store NVIDIA CCluster tokens and other vault objects. + Learn how to generate and store NVIDIA CCluster tokens and other secrets. ` — show details for one deployment | | `pause` / `resume` / `delete` | `ccluster pause ` — change or remove a deployment | | `shell` | `ccluster shell [--pod NAME] [--shell bash\|sh\|zsh] [--first-pod]` — open an interactive shell in a running pod | @@ -73,11 +73,11 @@ centml logout - Learn how to generate and store NVIDIA CCluster tokens and other vault objects. + Learn how to generate and store NVIDIA CCluster tokens and other secrets. The first steps to help you get started with the NVIDIA CCluster. + + + The open-source CCluster project on GitHub. + diff --git a/docs/home/open-source.mdx b/docs/home/open-source.mdx new file mode 100644 index 0000000..0047d8c --- /dev/null +++ b/docs/home/open-source.mdx @@ -0,0 +1,35 @@ +--- +title: Open Source CCluster +description: The open-source CCluster project on GitHub +--- + +CCluster is also developed as an open-source project under the Apache 2.0 license at [github.com/NVIDIA/ccluster](https://github.com/NVIDIA/ccluster). It is a Kubernetes-native platform for managing AI workloads across distributed, multi-cloud GPU infrastructure. + + +The open-source project is in an early stage and changes quickly. See the repository README for its current status and scope. + + +## How the project is structured + +- **Control-plane operator** — orchestrates the platform. It registers clusters, decides what is deployed where and in what order, and collects status from each cluster in one place. +- **Infra-plane operator** — runs on each workload cluster. It checks that the cluster's dependencies are installed and healthy, and reports back. +- **Catalog** — the Helm charts the control plane deploys, such as the GPU Operator, cert-manager, Istio, and KEDA, along with inference, training, and compute workloads. + +## Where to go next + + + + Browse the source, report issues, and follow development. + + + Deploy your first workload with the NVIDIA CCluster console. + + diff --git a/docs/home/quickstart.mdx b/docs/home/quickstart.mdx index 1a260cb..ef8363a 100644 --- a/docs/home/quickstart.mdx +++ b/docs/home/quickstart.mdx @@ -3,7 +3,7 @@ title: Quickstart description: Get Started with the NVIDIA CCluster --- -NVIDIA CCluster provides dedicated LLM endpoints, general inference, and compute deployments — all backed by optimized infrastructure. +NVIDIA CCluster provides general inference, NVIDIA Dynamo LLM serving, compute instances, and batch jobs — all backed by optimized infrastructure. No need to worry about GPU provisioning, scaling, or maintenance — just log in and deploy. @@ -18,19 +18,20 @@ To get started, sign in to the NVIDIA CCluster console using the access details Once logged in, you will see the NVIDIA CCluster console home page, as shown below. - NVIDIA CCluster console home page + NVIDIA CCluster home page with Inference, Compute, Job, and Dynamo tiles and cluster capacity ## 2. Create a bearer token -Endpoints are never public — every request must authenticate with a Bearer token or a client certificate. To interact with NVIDIA CCluster endpoints programmatically, generate a Bearer Token by following the [Managing Vault Objects](/resources/vault) documentation. For all protection options, see [Securing Endpoints](/resources/private). +Endpoints are never public — every request must authenticate with a Bearer token or a client certificate. To interact with NVIDIA CCluster endpoints programmatically, generate a Bearer Token by following the [Managing Secrets](/resources/vault) documentation. For all protection options, see [Securing Endpoints](/resources/private). ## 3. Deploy your first model Choose the deployment type that fits your use case: -- **[LLM Serving](/apps/llm)** — Deploy dedicated, protected LLM endpoints tailored to your performance requirements and budget. +- **[NVIDIA Dynamo](/apps/dynamo)** — Deploy protected LLM endpoints with aggregated or disaggregated prefill/decode serving, autoscaling, and scale to zero. - **[General Inference](/apps/inference)** — Deploy custom containerized models on NVIDIA-managed infrastructure. -- **[Compute](/apps/compute)** — Provision GPU compute for training, fine-tuning, or batch workloads. +- **[Compute](/apps/compute)** — Provision GPU compute instances for training and fine-tuning. +- **[Batch Jobs](/apps/jobs)** — Run containerized workloads to completion. ## Additional support: billing, sales, and/or technical @@ -42,11 +43,11 @@ For access, billing, sales, or technical assistance, contact your NVIDIA represe - Explore dedicated, protected endpoints for production model deployments. + Serve LLMs with disaggregated prefill/decode, autoscaling, and scale to zero. - Explore dedicated, protected endpoints for production model and model infrastructure management. + Serve LLMs with disaggregated prefill/decode, autoscaling, and scale to zero. /`, so set **Add command** to run ``` - Config File entry.sh installing vllm[grpc] and launching the gRPC server, invoked from Add command + General Inference Advanced step with Add command and Config File fields -Every endpoint is protected — there are no public endpoints. Secure access with a Bearer token, an mTLS client certificate, or both. For mTLS, select **Make it a private endpoint?** and CCluster generates a TLS certificate that clients must present. See [Securing Endpoints](/resources/private) for details. +Every endpoint is protected — there are no public endpoints. Secure access with a Bearer token, an mTLS client certificate, or both. For mTLS, select **Private Endpoint** and CCluster generates a TLS certificate that clients must present. See [Securing Endpoints](/resources/private) for details. ## 2. Select the cluster and hardware @@ -77,7 +77,7 @@ Every endpoint is protected — there are no public endpoints. Secure access wit Choose the cluster and GPU instance that match your model and setup, then click **Deploy**. - Hardware instance selection + General Inference Hardware step with cluster and hardware cards ## 3. Call your gRPC endpoint @@ -138,7 +138,7 @@ Endpoints are protected, so authenticate every call — pass a Bearer token, or Interact with NVIDIA CCluster programmatically. diff --git a/docs/resources/hardware-instances.mdx b/docs/resources/hardware-instances.mdx index 68a42e6..a5c01cc 100644 --- a/docs/resources/hardware-instances.mdx +++ b/docs/resources/hardware-instances.mdx @@ -1,6 +1,6 @@ --- title: 'Managing Hardware Instances' -description: 'Create, list, and delete hardware instances under a cluster with the Python SDK' +description: 'View cluster capacity and hardware in the console, and create, list, and delete hardware instances with the Python SDK' --- A **hardware instance** describes a class of hardware — GPU type, GPU count, CPU, and memory — that is available for deployments within a [cluster](/apps/compute). The [Python SDK](/clients/sdk) lets you list the hardware instances you have access to, and (with the right permissions) create and delete them. @@ -9,6 +9,44 @@ A **hardware instance** describes a class of hardware — GPU type, GPU count, C Creating and deleting hardware instances requires admin privileges (`PERM_ADMIN_MANAGE_HARDWARE`) on your NVIDIA CCluster organization. Listing hardware instances does not require admin privileges. +## View clusters and capacity in the console + +Select **Clusters** from the sidebar to see every cluster available to your organization. Each cluster is a collapsible row that shows: + +- **GPU capacity** — one chip per GPU type, such as `H200: 3/8 in use`. +- **Hardware configurations** — a table of the hardware instances you can deploy on the cluster, with **Name**, **GPU** (count and type), **GPU memory**, **CPU (cores)**, and **Memory**. + + + Clusters page showing GPU capacity chips and the hardware configurations table for a cluster + + +The **Accelerator Capacity** table on the console home page shows the same capacity data, with a usage bar per GPU type. The bar is green below 70%, amber from 70% to 89%, and red at 90% or more. + + +**In use** counts the GPUs that running workloads have requested, not how busy those GPUs are. A cluster whose capacity cannot be read shows "Capacity data is not currently available". + + +When you create a deployment, the **Hardware** step shows one card per GPU type for the cluster you select, with its GPU memory, system memory, CPU count, and the GPU counts (1x, 2x, 4x, 8x) that are available. Cards for GPU types that the selected cluster does not offer are greyed out. If a cluster is closed to new deployments, creating one fails with "Cluster does not allow new deployments". + +### Add or remove hardware configurations + +Administrators can manage a cluster's hardware catalog from the same page. Click **Add hardware configuration** in a cluster and fill in: + +- **GPU type** — chosen from the shared hardware catalog. It sets the GPU memory shown for the configuration. +- **Hardware name** — for example `x4-large-b200`. +- **GPU count** — `1`, `2`, `4`, or `8`. +- **CPU (millicores)** — `1000` millicores is one CPU core. +- **Memory (MiB)** — `1024` MiB is one GiB. +- **Node affinity labels** — at least one key and value that the scheduler must match when it places the hardware. Selecting a GPU type pre-fills `nvidia.com/gpu.product`. + +To remove a configuration, click the trash button in its row and confirm **Delete**. This removes the configuration from the cluster catalog so new deployments cannot use it. Existing deployments are not deleted. + +Adding and deleting hardware requires the `PERM_ADMIN_MANAGE_HARDWARE` permission. The **Add hardware configuration** button is visible to everyone in the console, but the request is rejected for users without that permission. + +## Manage hardware with the Python SDK + +The rest of this page shows how to list, create, and delete hardware instances with the [Python SDK](/clients/sdk). + ## Prerequisites Install the SDK and authenticate before running any of the examples below. For unattended use in CI or automation, authenticate with a [service account](/resources/service-accounts) by setting `CENTML_SERVICE_ACCOUNT_ID` and `CENTML_SERVICE_ACCOUNT_SECRET`. Interactive development can use `centml login` instead — see [Client Setup](/clients/setup). diff --git a/docs/resources/images/add-bearer-token.png b/docs/resources/images/add-bearer-token.png index 033e521..82b8ab8 100644 Binary files a/docs/resources/images/add-bearer-token.png and b/docs/resources/images/add-bearer-token.png differ diff --git a/docs/resources/images/add-certificates.png b/docs/resources/images/add-certificates.png index c57bf49..2265cfa 100644 Binary files a/docs/resources/images/add-certificates.png and b/docs/resources/images/add-certificates.png differ diff --git a/docs/resources/images/add-env.png b/docs/resources/images/add-env.png index 3e95119..f040cee 100644 Binary files a/docs/resources/images/add-env.png and b/docs/resources/images/add-env.png differ diff --git a/docs/resources/images/added-bearer-token.png b/docs/resources/images/added-bearer-token.png index 4bb0063..a408cd7 100644 Binary files a/docs/resources/images/added-bearer-token.png and b/docs/resources/images/added-bearer-token.png differ diff --git a/docs/resources/images/added-certificate.png b/docs/resources/images/added-certificate.png index 58bc787..a408cd7 100644 Binary files a/docs/resources/images/added-certificate.png and b/docs/resources/images/added-certificate.png differ diff --git a/docs/resources/images/clusters.png b/docs/resources/images/clusters.png new file mode 100644 index 0000000..0428bc9 Binary files /dev/null and b/docs/resources/images/clusters.png differ diff --git a/docs/resources/images/env-vault.png b/docs/resources/images/env-vault.png index 9c4510d..a408cd7 100644 Binary files a/docs/resources/images/env-vault.png and b/docs/resources/images/env-vault.png differ diff --git a/docs/resources/images/gen-cert.png b/docs/resources/images/gen-cert.png index 4458c7d..911ec2f 100644 Binary files a/docs/resources/images/gen-cert.png and b/docs/resources/images/gen-cert.png differ diff --git a/docs/resources/images/grpc-vllm-config.png b/docs/resources/images/grpc-vllm-config.png index 82274e0..dc8d8b1 100644 Binary files a/docs/resources/images/grpc-vllm-config.png and b/docs/resources/images/grpc-vllm-config.png differ diff --git a/docs/resources/images/grpc-vllm-entry-script.png b/docs/resources/images/grpc-vllm-entry-script.png index 0b8c672..04c4876 100644 Binary files a/docs/resources/images/grpc-vllm-entry-script.png and b/docs/resources/images/grpc-vllm-entry-script.png differ diff --git a/docs/resources/images/grpc-vllm-hardware.png b/docs/resources/images/grpc-vllm-hardware.png index 9f3534d..84ccd7e 100644 Binary files a/docs/resources/images/grpc-vllm-hardware.png and b/docs/resources/images/grpc-vllm-hardware.png differ diff --git a/docs/resources/images/storage-create-s3.png b/docs/resources/images/storage-create-s3.png new file mode 100644 index 0000000..d6b859f Binary files /dev/null and b/docs/resources/images/storage-create-s3.png differ diff --git a/docs/resources/images/storage-create.png b/docs/resources/images/storage-create.png new file mode 100644 index 0000000..53f41f8 Binary files /dev/null and b/docs/resources/images/storage-create.png differ diff --git a/docs/resources/images/storage-list.png b/docs/resources/images/storage-list.png new file mode 100644 index 0000000..770303b Binary files /dev/null and b/docs/resources/images/storage-list.png differ diff --git a/docs/resources/images/vault-bearer-tokens.png b/docs/resources/images/vault-bearer-tokens.png index 6d0538a..e8a5bdf 100644 Binary files a/docs/resources/images/vault-bearer-tokens.png and b/docs/resources/images/vault-bearer-tokens.png differ diff --git a/docs/resources/images/vault.png b/docs/resources/images/vault.png index 58002a7..a408cd7 100644 Binary files a/docs/resources/images/vault.png and b/docs/resources/images/vault.png differ diff --git a/docs/resources/json-and-tool.mdx b/docs/resources/json-and-tool.mdx index 6dab6f0..788ced6 100644 --- a/docs/resources/json-and-tool.mdx +++ b/docs/resources/json-and-tool.mdx @@ -3,13 +3,13 @@ title: 'Agents on NVIDIA CCluster' description: 'How to build AI agents using structured outputs such as JSON and function/tool calls with NVIDIA CCluster' --- -NVIDIA CCluster's **Dedicated** LLM APIs support structured outputs using the same OpenAI-compatible API. These features are crucial for agentic workloads that require reliable data parsing and function calling. The NVIDIA CCluster also provides reasoning-enabled models (e.g., `DeepSeek-AI/deepseek-r1`) that can perform reasoning before generating structured outputs. +LLM endpoints you deploy on NVIDIA CCluster, such as [NVIDIA Dynamo](/apps/dynamo), expose an OpenAI-compatible API. Structured outputs such as JSON schema constraints and function or tool calls are crucial for agentic workloads that require reliable data parsing. Whether a given structured-output or tool-calling feature works depends on the model you deploy and the engine settings you choose, so test it against your own endpoint. Reasoning models, which can reason before generating structured outputs, work the same way. ## JSON schema output When you need a response strictly formatted as JSON, you can use JSON schema constraints. This is particularly useful in scenarios where your system or other downstream processes rely on valid JSON. -Below is an example of how to set up a request to the NVIDIA CCluster LLM API using JSON schema enforcement. Replace `` with the endpoint URL of your [LLM Serving](/apps/llm) deployment, and set `CENTML_API_KEY` to a bearer token from your [Vault](/resources/vault). +Below is an example of how to set up a request to the NVIDIA CCluster LLM API using JSON schema enforcement. Replace `` with the endpoint URL of your [NVIDIA Dynamo](/apps/dynamo) deployment (or another OpenAI-compatible [General Inference](/apps/inference) endpoint), and set `CENTML_API_KEY` to a bearer token from your [Secrets](/resources/vault). ```python from openai import OpenAI @@ -18,7 +18,7 @@ import os # Retrieve the model name from environment variables model = os.environ.get("MODEL", "deepseek-ai/DeepSeek-R1") -# Supply your API key from your NVIDIA CCluster vault +# Supply your API key from your NVIDIA CCluster Secrets api_key = os.environ.get("CENTML_API_KEY") assert api_key is not None, "Please provide an API Key" @@ -62,7 +62,7 @@ schema_json = { # Initialize the OpenAI-compatible client with the API key and base URL client = OpenAI( api_key=api_key, - base_url="https:///openai/v1", + base_url="https:///v1", ) # Create a chat completion request with the defined prompt and schema @@ -102,10 +102,10 @@ import os import json import time -# Supply your API key from your NVIDIA CCluster vault +# Supply your API key from your NVIDIA CCluster Secrets api_key = os.environ.get("CENTML_API_KEY") assert api_key is not None, "Please provide an API Key" -base_url = "https:///openai/v1" +base_url = "https:///v1" def get_weather(location): """ @@ -327,7 +327,7 @@ For more details, continue exploring this documentation set or use your establis Learn how to build your AI application using NVIDIA CCluster diff --git a/docs/resources/managing-deployments.mdx b/docs/resources/managing-deployments.mdx index c7cde9d..6117bc0 100644 --- a/docs/resources/managing-deployments.mdx +++ b/docs/resources/managing-deployments.mdx @@ -5,7 +5,7 @@ description: 'Operate deployments from the console: logs, in-browser Terminal, p After you create a deployment, open it from the deployments list to inspect status, stream logs, open a shell, pause or resume it, clone it, or (for revision-aware types) roll back to an earlier spec. -This page covers those shared operations. Create-form fields stay on the type pages: [LLM Serving](/apps/llm), [General Inference](/apps/inference), [NVIDIA Dynamo](/apps/dynamo), [Compute](/apps/compute), and [Batch Jobs](/apps/jobs). +This page covers those shared operations. Create-form fields stay on the type pages: [General Inference](/apps/inference), [NVIDIA Dynamo](/apps/dynamo), [Compute](/apps/compute), and [Batch Jobs](/apps/jobs). ## Detail page tabs @@ -40,11 +40,11 @@ For Compute instances, the Terminal does **not** replace SSH. Use [SSH](/apps/co ## Pause and resume -On the deployment detail page you can **Pause** an Active deployment or **Resume** a paused one. Pausing stops serving (or stops the Compute instance) without deleting the deployment. +On the deployment detail page you can **Pause** an Active deployment or **Resume** a paused one. Pausing stops serving (or stops the Compute instance) without deleting the deployment. On Compute, pausing deletes files on the default disk unless you mounted a writable volume at `/home` or `/root`; files in a mounted volume survive pause. Restarts and updates keep your home files. -- **LLM Serving**, **General Inference**, **Compute**: pause and resume the whole deployment. +- **General Inference** and **Compute**: pause and resume the whole deployment. - **Dynamo**: pause and resume the whole deployment. You cannot pause only an in-progress Dynamo rollout. See [NVIDIA Dynamo](/apps/dynamo). -- **Jobs**: jobs run to completion and then stop. Pause and resume do not apply. +- **Jobs**: pause and resume the whole job from the details page. Pausing deletes the job's pods and resuming restarts the job from the beginning, so completed work can run again. Checkpoint in your application and write to persistent storage to keep progress. See [Batch Jobs](/apps/jobs) and [Managing Storage](/resources/storage). You can also pause and resume from the CLI (`ccluster pause ` / `ccluster resume `). See [Client Setup](/clients/setup). @@ -52,13 +52,13 @@ You can also pause and resume from the CLI (`ccluster pause ` / `ccluster re **Clone** opens the create form prefilled from an existing deployment (the name typically gets a `-clone` suffix). Review hardware, secrets, and tokens before you deploy the copy — cloned credentials such as Hugging Face tokens may need to be selected again. -**Edit** (update) changes the running deployment. On revision-aware types, spec edits create a **new revision** and start a rollout. Compute and jobs are not revision-aware in the console: an edit updates the deployment without a revision table or rollback UI. +**Edit** (update) changes the running deployment. On revision-aware types, spec edits create a **new revision** and start a rollout. Jobs are not revision-aware in the console and cannot be edited after creation. ## Revisions, rollback, and rollout -**LLM Serving**, **General Inference**, and **Dynamo** are revision-aware. The **Detail** tab lists revisions so you can inspect status and logs per revision and **roll back** to an earlier revision. +**General Inference**, **Dynamo**, and **Compute** are revision-aware. The **Detail** tab lists revisions so you can inspect status and logs per revision and **roll back** to an earlier revision. -**Compute** and **Jobs** do not expose a revision table or rollback in the console. +**Jobs** do not expose a revision table or rollback in the console. When you edit a revision-aware deployment that is already Active, you can set rollout surge: @@ -89,16 +89,16 @@ On Dynamo, pick a Frontend, Worker, Prefill, or Decode pod; replica counts are r ## What each deployment type supports -| Action | LLM Serving | General Inference | Dynamo | Compute | Jobs | -| --- | --- | --- | --- | --- | --- | -| Logs | Yes, if logging is enabled | Yes, if logging is enabled | Yes, if logging is enabled | Yes, if logging is enabled | Yes, if logging is enabled | -| Terminal | While **Active** | While **Active** | While **Active** | While **Active** | While pods are running | -| Pause / resume | Yes | Yes | Whole deployment only | Yes | Not applicable | -| Clone | Yes | Yes | Yes | Yes | Yes | -| Edit | New revision | New revision | New revision | In-place update | In-place update | -| Revision table / rollback | Yes | Yes | Yes | No | No | -| Change rollout (`max_surge` / `max_unavailable`) | Yes | Yes | Aggregated deployments | No | No | -| Scale replicas without a new revision | Yes | Yes | Aggregated Workers | Not a multi-replica serving deployment | Not applicable | +| Action | General Inference | Dynamo | Compute | Jobs | +| --- | --- | --- | --- | --- | +| Logs | Yes, if logging is enabled | Yes, if logging is enabled | Yes, if logging is enabled | Yes, if logging is enabled | +| Terminal | While **Active** | While **Active** | While **Active** | While pods are running | +| Pause / resume | Yes | Whole deployment only | Yes | Yes | +| Clone | Yes | Yes | Yes | Yes | +| Edit | New revision | New revision | New revision | Not editable | +| Revision table / rollback | Yes | Yes | Yes | No | +| Change rollout (`max_surge` / `max_unavailable`) | Yes | Aggregated deployments | No | No | +| Scale replicas without a new revision | Yes | Aggregated Workers | Not a multi-replica serving deployment | Not applicable | ## What's next @@ -111,11 +111,11 @@ On Dynamo, pick a Frontend, Worker, Prefill, or Decode pod; replica counts are r Configure logging, affinity, caching, config files, and autoscaling when you create an endpoint. - Deploy a dedicated LLM endpoint, then operate it from the detail page. + Deploy a Dynamo endpoint, then operate it from the detail page. - Explore dedicated, protected endpoints for production model deployments. + Serve LLMs with disaggregated prefill/decode, autoscaling, and scale to zero. /v1/chat/completions \ ``` -Every endpoint is protected — there are no public endpoints. Secure access with a Bearer token, an mTLS client certificate, or both. For mTLS, select **Make it a private endpoint?** and CCluster generates a TLS certificate that clients must present. See [Securing Endpoints](/resources/private) for details. +Every endpoint is protected — there are no public endpoints. Secure access with a Bearer token, an mTLS client certificate, or both. For mTLS, select **Private Endpoint** and CCluster generates a TLS certificate that clients must present. See [Securing Endpoints](/resources/private) for details. ## What's next @@ -84,7 +85,7 @@ Every endpoint is protected — there are no public endpoints. Secure access wit Protect your endpoints with Bearer tokens and mTLS certificates. diff --git a/docs/resources/private.mdx b/docs/resources/private.mdx index 8dff3a0..19b031f 100644 --- a/docs/resources/private.mdx +++ b/docs/resources/private.mdx @@ -3,24 +3,26 @@ title: 'Securing Endpoints' description: 'How to protect your inference endpoints with Bearer tokens and mTLS certificates' --- -NVIDIA CCluster does not expose public endpoints. Every LLM Serving and General Inference deployment must be protected, and clients must authenticate on each request. You secure a deployment with **at least one** of the following: +NVIDIA CCluster does not expose public endpoints. Every General Inference and Dynamo deployment must be protected, and clients must authenticate on each request. You secure a deployment with **at least one** of the following: - **Bearer token** — an `Authorization: Bearer ` header on each request. - **mTLS client certificate** — a TLS certificate the client presents on each request. -Endpoint bearer tokens (from your [Vault](/resources/vault)) are not the same as [service account](/resources/service-accounts) credentials. Service accounts authenticate automation to NVIDIA CCluster platform APIs; Vault bearer tokens protect calls to your deployed endpoints. +Endpoint bearer tokens (from your [Secrets](/resources/vault)) are not the same as [service account](/resources/service-accounts) credentials. Service accounts authenticate automation to NVIDIA CCluster platform APIs; bearer tokens from Secrets protect calls to your deployed endpoints. You choose the protection method when configuring a deployment. You may enable both for defense in depth. ## Bearer token access -Generate a Bearer token from your [Vault](/resources/vault), then include it in the `Authorization` header of every request. +Generate a Bearer token from your [Secrets](/resources/vault), then include it in the `Authorization` header of every request. + +The request path comes from the server running in your deployment. The examples use `/v1/chat/completions`, which OpenAI-compatible servers such as vLLM, TensorRT-LLM, NIM, and Dynamo expose. If your container serves a different API, use its path. **Using curl command** ```bash showLineNumbers={false} -curl -X 'POST' 'https:///openai/v1/chat/completions' \ +curl -X 'POST' 'https:///v1/chat/completions' \ -H 'Authorization: Bearer ' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ @@ -42,19 +44,20 @@ from openai import OpenAI client = OpenAI( api_key="", - base_url="https:///openai/v1", + base_url="https:///v1", ) ``` ## mTLS certificate access -Select the Make it a private endpoint? option when configuring a deployment. This generates a TLS certificate upon deployment, which you can download. Access to the endpoint will then require the http client to use this certificate. Here are a few examples: +When configuring a deployment, under **Private Endpoint**, choose **Certificate**. Then select an existing certificate, or select **+ Add a new certificate** to generate one or use your own. When you generate a new certificate, its `.pem` file downloads to your browser at that moment, and the certificate is saved to your [Secrets](/resources/vault). Access to the endpoint then requires the HTTP client to present this certificate. Here are a few examples: **Using curl command** ```bash showLineNumbers={false} -curl -X 'POST' 'https:///openai/v1/chat/completions' \ - --cert \ # Downloaded certificate +# --cert points to the .pem file downloaded when you generated the certificate +curl -X 'POST' 'https:///v1/chat/completions' \ + --cert \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ @@ -93,7 +96,7 @@ from openai import OpenAI client = OpenAI( api_key="no_key", - base_url="https:///openai/v1", + base_url="https:///v1", http_client=httpx.Client(cert=""), ) ``` @@ -103,11 +106,11 @@ client = OpenAI( - Learn how to generate and store NVIDIA CCluster tokens and other vault objects. + Learn how to generate and store NVIDIA CCluster tokens and other secrets. -Service accounts authenticate to **NVIDIA CCluster platform APIs** (for example, listing deployments or creating workloads through the SDK). They are separate from [Vault bearer tokens](/resources/vault), which protect your deployed inference endpoints. See [Securing Endpoints](/resources/private) for endpoint access. +Service accounts authenticate to **NVIDIA CCluster platform APIs** (for example, listing deployments or creating workloads through the SDK). They are separate from [bearer tokens from Secrets](/resources/vault), which protect your deployed inference endpoints. See [Securing Endpoints](/resources/private) for endpoint access. ## Prerequisites @@ -115,7 +115,7 @@ API requests made with a service account are attributed to the user who created Access NVIDIA CCluster programmatically using Python. diff --git a/docs/resources/storage.mdx b/docs/resources/storage.mdx new file mode 100644 index 0000000..f1301b4 --- /dev/null +++ b/docs/resources/storage.mdx @@ -0,0 +1,129 @@ +--- +title: 'Managing Storage' +description: 'Create filesystem volumes and object-storage connections, and mount them into your deployments' +--- + +**Storage** gives your deployments data that outlives them: model weights, datasets, checkpoints, and home directories. Storage is created once, belongs to one cluster, and can be mounted into [Compute](/apps/compute), [Batch Jobs](/apps/jobs), [General Inference](/apps/inference), and [NVIDIA Dynamo](/apps/dynamo) deployments on that cluster. + +There are two storage types: + +| Type | What it is | Good for | +| --- | --- | --- | +| **Filesystem** | A persistent volume you size yourself. Every deployment that mounts it shares the same files (`ReadWriteMany` access). | Datasets, checkpoints, model caches, persistent home directories | +| **Object storage** | A connection to an **existing** Amazon S3 bucket, mounted through the cluster's S3 CSI driver. | Reading large datasets that already live in a bucket | + + +S3 mounts do not provide normal Linux filesystem semantics. Use a **Filesystem** volume for workloads that need them. + + +## 1. Create storage + +Select **Storage** from the sidebar, then click **Create storage**. + + + Storage page listing volumes with name, cluster, type, provider, size or bucket, and status + + +Choose a **Storage type**, then fill in the fields. + +### Filesystem + + + Create storage dialog for a filesystem volume + + +- **Name** — up to 20 characters: lowercase letters, numbers, and hyphens, starting with a letter. A name must be unique per user and cluster. +- **Cluster** — the cluster that hosts the volume. A volume can only be mounted by deployments on this cluster. +- **Size (GiB)** — a whole number from `1` to `65536`. The default is `100`. +- **Storage class** — optional. Leave it blank to use the cluster's configured storage class. + +### Object storage + + + Create storage dialog for an S3 object-storage connection + + +- **Name** and **Cluster** — as above. +- **Provider** — `S3`. +- **Bucket** — the name or ARN of an existing bucket. NVIDIA CCluster does not create the bucket. +- **Region** — the bucket's region, for example `us-east-1`. +- **Prefix** — optional object-key prefix that ends in `/`. Leave it blank to mount the whole bucket. +- **Read-only** — turn on to make every mount of this connection read-only. + +No cloud credentials are entered in the console. The cluster's S3 CSI driver must already be configured with an identity that can access the bucket, so ask your cluster administrator if mounts cannot read the bucket. + + +You cannot resize or edit a volume after you create it, and the storage class is fixed. Create a new volume if you need a different size or class. + + +## 2. Check the status + +The **Status** column shows the live state of the storage: + +| Status | Meaning | +| --- | --- | +| **Healthy** | The storage is provisioned and bound. | +| **Initializing** | Provisioning is not finished. Some storage classes wait for the first deployment that mounts the volume, so this can be normal until then. You can still select the volume. | +| **Error** | Provisioning failed or the storage was lost. Open the volume for the error message, then delete it and create it again. | +| **Missing** | The storage resources are missing from the cluster. Delete the volume and create it again. | +| **Unknown** | The status could not be determined. | + +Click a row to open the volume details: cluster, size and storage class (or provider, bucket, region, prefix, and access), access mode, ID, and creation time. Use **Filter by cluster** to narrow the list. The list shows storage for your whole organization, but you can only mount volumes that you created. + +## 3. Mount storage in a deployment + +The **Hardware** step of the Compute, Job, Inference, and Dynamo create and edit forms has a **Storage** section. It appears after you select a cluster. Click **Add mount** and set: + +- **Volume** — an active volume on the same cluster. If none are listed, create storage first. +- **Mount path** — the absolute path in the container, for example `/data`. +- **Read-only** — mount the volume read-only. A read-only S3 connection is always mounted read-only. + +Rules for mounts: + +- A deployment can have up to **10** mounts, and each volume can be mounted **once** per deployment. +- Mount paths must be absolute, unique, not `/`, and must not contain `..`. Paths that the platform uses itself, such as `/dev/shm`, are rejected. +- You can only mount volumes you created, on the deployment's own cluster. If you change the cluster, mounts for other clusters are removed. +- Mounts apply to the compute instance, to every pod of a job, to the inference application container, or to the Dynamo Frontend and every worker. +- When you edit a deployment and remove a mount, only the new revision changes. The volume and its data are kept. + +### Keep your Compute home directory + +A Compute instance's SSH home folder is `/root`. Files in it survive restarts and updates, but pausing the instance deletes files on the default disk. To keep them across a pause, mount a **writable** volume at `/home` (or at `/root`). A volume mounted at `/home` gets its own sub-directory per deployment, and the files from the image are copied in the first time. A volume mounted directly at `/root` is used as it is. Changing volumes does not copy existing files. + +## 4. Delete storage + +Click the trash button on a row, and then **Delete**. + +- For a **Filesystem** volume, files may be permanently deleted, depending on the storage class's reclaim policy. Retention is not guaranteed. +- For **Object storage**, only the mount resources are removed from the cluster. Objects in the bucket are left intact. +- You can delete storage that you created. Organization administrators with the volume-management permission can delete any volume. + + +You cannot delete storage while any retained revision of a deployment that still exists uses it. That includes older revisions, paused deployments, and completed jobs. Removing the mount from the current revision does not release older revisions. Delete those deployments first. + + +## What's next + + + + Persist your home directory across pauses. + + + Share datasets and results across job pods. + + + Pause, resume, edit, and roll back deployments. + + diff --git a/docs/resources/trtllm.mdx b/docs/resources/trtllm.mdx index 5fac72d..c0a4538 100644 --- a/docs/resources/trtllm.mdx +++ b/docs/resources/trtllm.mdx @@ -31,7 +31,7 @@ The image might appear as `nvidia/tensorrt-llm/release` in the prebuilt image li ### Add command -The TensorRT-LLM release image does not start an inference server by itself. Under **Optional Details → Add command**, enter: +The TensorRT-LLM release image does not start an inference server by itself. In the **Advanced** step, under **Add command**, enter: ```bash showLineNumbers={false} /bin/bash -lc 'trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --host 0.0.0.0 --port 8000 --config /etc/engine/config.yaml' @@ -78,7 +78,7 @@ Tune `max_batch_size`, `max_seq_len`, and `free_gpu_memory_fraction` for your mo ### Model credentials -Under **Optional Details → Environment Variables**, set `HF_TOKEN` to a Hugging Face token when the model is gated or private. You can store the token in a [Vault object](/resources/vault) instead of entering it directly. +In the **Endpoint** step, select a **Hugging Face Token** saved in [Secrets](/resources/vault) when the model is gated or private. Every endpoint must also be protected with a Bearer token, an mTLS client certificate, or both. See [Securing Endpoints](/resources/private). @@ -144,7 +144,7 @@ Use the exact model `id` returned by `/v1/models` in subsequent requests. The fi Protect your endpoints with Bearer tokens and mTLS certificates. diff --git a/docs/resources/usage.mdx b/docs/resources/usage.mdx new file mode 100644 index 0000000..018ce1e --- /dev/null +++ b/docs/resources/usage.mdx @@ -0,0 +1,95 @@ +--- +title: 'Monitoring Usage' +description: 'View request, latency, GPU, CPU, and memory metrics for your deployments in the console' +--- + +The **Usage** page shows live metrics for one deployment at a time. The same charts appear on the **Usage** tab of each deployment's detail page, with the deployment already selected. Usage is for monitoring performance and capacity. It does not show charges. + +## Open the metrics + +- Select **Usage** from the sidebar, then pick a deployment in **Deployments**. The newest deployment is selected by default. +- Or open a deployment from **Deployments** and select its **Usage** tab. + +If you have no deployments yet, the page shows **No deployments found** and a **New Deployment** button. + +## Choose a time range + +Use **Time range** to pick **Last 30 mins**, **Last 1 hour** (the default), **Last 3 hours**, **Last 6 hours**, or **Custom**. A custom range uses a start and end date, can reach back at most one month, and cannot be longer than one month. A single day set to today runs from midnight to now. + +Click the refresh button to reload the charts. Relative ranges are recomputed from the current time when you refresh. The charts pick their step size from the length of the range, and times are shown in your browser's time zone. If a deployment stopped reporting for a while, the chart shows a gap. A range that ends before the deployment was created is empty. + +## Charts for General Inference, Compute, and Jobs + +**Deployment & Request Metrics** appears for inference deployments: + +| Chart | What it shows | +| --- | --- | +| **HTTP Requests** | Number of requests per step. | +| **Latency** | Response time in seconds, as p50, p90, and p99. | +| **HTTP Requests by Status** | Requests per HTTP status code. | +| **Error Codes** | Total 4xx and 5xx responses in the selected range, per status code. | + +**Hardware Resource Utilization by Pod** appears for every deployment type, including jobs. Use **Revision** and **Pod** to choose which pod to inspect. + +| Chart | What it shows | +| --- | --- | +| **GPU** | GPU utilization, 0–100%, per GPU. | +| **GPU Memory** | GPU memory in use, with the GPU's memory as the maximum. | +| **CPU** | CPU use as a percentage of the container's CPU limit. | +| **Memory** | Container memory in use, with the hardware's memory as the maximum. | + +## Charts for NVIDIA Dynamo + +Dynamo deployments have a dedicated view. Use **Metrics** to switch between views, and **Revision** to pick **All revisions** or a single revision. It defaults to the current revision. + +- **Overview** + - **Frontend TTFT** — time to first token, in milliseconds, as p50, p95, and p99. + - **Time per output token** — milliseconds per token. + - **Request duration** — seconds, without client network latency. + - **Output throughput** — tokens per second. + - **Request outcomes** — requests per second by result. Cancellations are shown separately. + - **Queue depth** — queued requests. Prefill and Decode are shown separately. +- **Workers** — choose a **Worker pool** (**Worker**, **Prefill**, or **Decode**) and a **Worker**, then see **GPU utilization**, **GPU memory**, **Running requests**, and **KV cache occupancy**. **CPU and container memory** is in a collapsible section. +- **KV transfer** — shown only for disaggregated deployments: **KV transfer throughput**, **KV transfer duration**, **Transfer completion rate**, and **Transfer failure rate**. + + +KV transfer charts show how much KV cache moved and how long it took. They do not tell you which transport carried it, so they cannot confirm that GPU Direct RDMA was used. + + +## Empty charts + +| Message | Meaning | +| --- | --- | +| **No observations in this period** | The deployment received no traffic in the range. | +| **No samples in this period** | The range is before the deployment existed, or no data was collected. | +| **Metrics are not available for this selection yet.** | The metric has not been reported for this deployment or pod. | +| **Metrics stopped reporting. Showing the last available samples.** | The deployment stopped sending metrics. The last sample time is shown. | +| **Some workers are not reporting metrics. Showing available samples.** | Some Dynamo workers are missing from the chart. | + +Use **Retry** on a chart that shows **Failed to load**. + +## What's next + + + + Check logs, pause, edit, and roll back from the detail page. + + + Learn about aggregated and disaggregated serving. + + + See cluster capacity and available hardware. + + diff --git a/docs/resources/vault.mdx b/docs/resources/vault.mdx index 2b8d7dd..8d06e90 100644 --- a/docs/resources/vault.mdx +++ b/docs/resources/vault.mdx @@ -1,70 +1,70 @@ --- -title: 'Managing Vault Objects' -description: 'How to generate and store NVIDIA CCluster tokens and other vault objects' +title: 'Managing Secrets' +description: 'How to create and manage bearer tokens, certificates, registry credentials, and other secrets in NVIDIA CCluster' --- -The NVIDIA CCluster allows you to generate and manage `Vault` objects such as `Bearer Tokens`, `Certificates`, `Registry Credentials`, `Environment Variables`, `Hugging Face Tokens`, and `SSH Keys`. -This guide will walk you through creating `Bearer Tokens` and other Vault objects you can use to configure or access NVIDIA CCluster services. +The **Secrets** page in the NVIDIA CCluster console lets you create and manage `Bearer Tokens`, `Certificates`, `Registry Credentials`, `Environment Variables`, `Hugging Face Tokens`, and `SSH keys`. These were previously called Vault objects. +This guide will walk you through creating `Bearer Tokens` and other secrets you can use to configure or access NVIDIA CCluster services. -Vault **bearer tokens** protect your deployed inference endpoints — clients include them in requests to LLM Serving and General Inference URLs. For **platform API automation** (SDK, CLI, deployment management without browser login), use [Service Accounts](/resources/service-accounts) instead. +Bearer tokens saved in Secrets protect your deployed inference endpoints — clients include them in requests to your General Inference and Dynamo endpoint URLs. For **platform API automation** (SDK, CLI, deployment management without browser login), use [Service Accounts](/resources/service-accounts) instead. ## Step 1: Log in to the NVIDIA CCluster -You will need to log in to the NVIDIA CCluster to manage Vault objects. If you do not yet have access, contact your NVIDIA representative or your organization's designated support contact before proceeding. +You will need to log in to the NVIDIA CCluster to manage secrets. If you do not yet have access, contact your NVIDIA representative or your organization's designated support contact before proceeding. -Once logged in, you will need to select `Account` from sidebar menu, and then click on the `Vault` tab from the `Your Account` window. +Once logged in, select **Secrets** from the sidebar menu. - Vault section of the account page + Secrets page -Once completed, move onto the second step below based on the type of Vault objects you are creating. +Once completed, move onto the second step below based on the type of secret you are creating. ## Step 2A: Creating bearer tokens Bearer tokens can be used to access NVIDIA CCluster services. -To generate a bearer token, select green `Add Vault Item`. A dropdown menu will appear. From that menu, select the `Bearer Tokens` option. +To generate a bearer token, select **Add Secret**. A dropdown menu will appear. From that menu, select the `Bearer Tokens` option. - Add Vault Item menu showing Bearer Tokens option + Add Secret menu showing Bearer Tokens option -A windows will pop up name `Add Bearer Token to Your Vault`. From there, you can either name and automatically generate a token with the `Generate new Bearer Token` option or you can enter a previous token using the `Use an existing Bearer Token` option. +A window named `Add Bearer Token to Your Secrets` will pop up. From there, you can either name and automatically generate a token with the `Generate new Bearer Token` option or you can enter a previous token using the `Use an existing Bearer Token` option. Form for adding a new bearer token to the vault -Once you either enter your token in the text box or opt into generating your token, click the green `Add to Vault` option and your Bearer Token should appear in your vault. +Once you either enter your token in the text box or opt into generating your token, choose a **Visibility** (see below) and click **Add to Secrets**. Your bearer token appears in the **Bearer Tokens** list. - Bearer token listed in the vault after creation + Bearer token listed in Secrets after creation -## Step 2B: Adding certificates to your vault +## Step 2B: Adding certificates NVIDIA CCluster uses client certificates for [mutual TLS (mTLS)](https://www.cloudflare.com/learning/access-management/what-is-mutual-tls/) -You can generate or add public certificates for mTLS just like you can with Bearer Tokens. Once you generate the certificate, it will appear in the vault. +You can generate or add public certificates for mTLS just like you can with Bearer Tokens. Once you generate the certificate, it appears in the **Certificates** list. - Add Vault Item menu showing Certificates option + Add Secret menu showing Certificates option - Certificate generation form in the vault + Certificate generation form - Generated certificate listed in the vault + Generated certificate listed in Secrets -When generated, `Vault Certificates` download a `.pem` file to your local browser. That `.pem` file is named after the generated cert. You can then use that `.pem` file to access one of NVIDIA CCluster's [secured endpoints](/resources/private) as long as they are associated with the appropriate certificate and `.pem` pair. +When generated, certificates download a `.pem` file to your local browser. That `.pem` file is named after the generated cert. You can then use that `.pem` file to access one of NVIDIA CCluster's [secured endpoints](/resources/private) as long as they are associated with the appropriate certificate and `.pem` pair. @@ -75,7 +75,7 @@ When generated, `Vault Certificates` download a `.pem` file to your local browse ### Example usage ```bash showLineNumbers={false} -curl -X POST 'https://centml-private-2.fe178792.c-09.centml.com/openai/v1/chat/completions' \ +curl -X POST 'https:///v1/chat/completions' \ --cert gen-cert.pem \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ @@ -101,42 +101,58 @@ curl -X POST 'https://centml-private-2.fe178792.c-09.centml.com/openai/v1/chat/c ``` ### Adding your own certificates -This section is currently under construction as we work to improve the UX around adding certificates to endpoints on the NVIDIA CCluster. Please check back soon! +If you already have a certificate authority (CA), you can use it instead of generating a certificate. + +1. On the **Secrets** page, select **Add Secret** and choose `Certificates`. +2. Select **Use an existing Certificate**. +3. Enter a name and paste the full certificate value (PEM format) of your CA into **Certificate Value**. +4. Select **Add to Secrets**. + +The certificate then appears in the **Certificates** list. Select it under **Private Endpoint** when you create a deployment, and clients that present a certificate signed by your CA can reach the endpoint. No `.pem` file is downloaded, because you already hold the client certificate and key. ## Step 2C: Adding registry credentials -Registry credentials allow the NVIDIA CCluster to pull container images from private registries when creating [General Inference](/apps/inference) or [Compute](/apps/compute) deployments. +Registry credentials allow the NVIDIA CCluster to pull container images from private registries when creating [General Inference](/apps/inference), [Compute](/apps/compute), or [Batch Jobs](/apps/jobs) deployments. -To add registry credentials, select the green `Add Vault Item` button and choose `Registry Credentials` from the dropdown. Enter a name, along with the registry username and password (or access token). The credentials are stored securely in your Vault and can be selected during deployment configuration. +To add registry credentials, select **Add Secret** and choose `Registry Credentials` from the dropdown. Enter a name, along with the registry username and password (or access token). The credentials are stored securely and can be selected as **Image Registry Credentials** during deployment configuration. Supported registries include Docker Hub, Amazon ECR, Google Artifact Registry, Azure Container Registry, and any Docker-compatible private registry. CCluster automatically detects the registry from the image URL at deployment time. ## Step 2D: Adding environment variables +Select **Add Secret**, choose `Environment Variables`, and enter the variable name and value. When you configure a deployment, you can select a saved environment variable instead of typing its value. + - Form for adding an environment variable to the vault + Form for adding an environment variable - Environment variable listed in the vault + Environment variable listed in Secrets +## Visibility and lifecycle + +- **Visibility** — each secret is visible either to you (`User`) or to your organization (`Organization`). The Secrets table shows the visibility of each secret. A secret name must be unique per type and visibility. +- **Secrets are immutable** — you cannot edit a secret after you create it. To rotate a value, create a new secret and update your deployments to use it, then delete the old one. +- **Deleting is blocked while in use** — you cannot delete a secret that the selected revision of any deployment still references, including paused deployments. Switch those deployments to another secret first. +- **Deployments reference secrets** — deployments store a reference to the secret rather than a copy of its value, so you pick Hugging Face tokens, registry credentials, bearer tokens, and environment variables from Secrets when you configure a deployment. + ## What's next - Explore dedicated, protected endpoints for production model deployments. + Deploy protected endpoints for your own containers and models.