Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# NVIDIA CCluster Docs

This repository contains the [Fern](https://buildwithfern.com/learn/docs/getting-started/overview) source for the NVIDIA CCluster documentation site, published at `docs.nvidia.com/ccluster`.
This repository contains the [Fern](https://buildwithfern.com/learn/docs/getting-started/overview) source for the NVIDIA CCluster documentation site, published at [docs.nvidia.com/ccluster](https://docs.nvidia.com/ccluster).

The site uses the NVIDIA global theme (`global-theme: nvidia`), so logo, colors, fonts, layout, and footer come from the shared [global-theme](https://gitlab-master.nvidia.com/fern-documentation/global-theme) repository rather than this repo. See the [NVIDIA Fern onboarding guide](https://gitlab-master.nvidia.com/fern-documentation/guides) for platform details.

Expand Down Expand Up @@ -52,7 +52,7 @@ python3 tools/check-docs-paths --allow '../docs/' docs fern
2. Add new pages to the `navigation` section of [fern/docs.yml](fern/docs.yml) with a `../docs/<section>/<page>.mdx` path; only pages listed there are published.
3. Put images in the section's `images/` folder and reference them with `./images/...` paths. Absolute paths and `..` segments are rejected by `tools/check-docs-paths`; the only exception is the `../docs/` prefix of page paths in `fern/docs.yml`.
4. Use Fern's [built-in components](https://buildwithfern.com/learn/docs/writing-content/components/overview) (`Card`, `CardGroup`, `Frame`, `Note`, `Tip`, and others). Icons are Font Awesome classes such as `fa-solid fa-lock`.
5. Link between pages with root-relative paths such as `/apps/llm`; Fern adds the `/ccluster` base path.
5. Link between pages with root-relative paths such as `/apps/dynamo`; Fern adds the `/ccluster` base path.

## Release Process

Expand All @@ -64,4 +64,4 @@ Publishing runs through GitHub Actions and needs the `FERN_TOKEN` repository sec
| [preview-docs.yml](.github/workflows/preview-docs.yml) | Same-repo PRs | Posts a Fern preview link on the PR |
| [publish-docs.yml](.github/workflows/publish-docs.yml) | Merge to `main` that changes `docs/` or `fern/`, or manual run | Publishes production after the checks pass |

Connecting the Fern site to `docs.nvidia.com/ccluster` requires a review and Jira ticket from the Developer Experience team; ask in [#cdd-fern](https://nvidia.enterprise.slack.com/archives/C0A8987CQ7K).
Merged changes go live at [docs.nvidia.com/ccluster](https://docs.nvidia.com/ccluster), the custom domain configured in [fern/docs.yml](fern/docs.yml). For Fern platform or domain issues, ask in [#cdd-fern](https://nvidia.enterprise.slack.com/archives/C0A8987CQ7K).
21 changes: 11 additions & 10 deletions docs/apps/compute.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,21 +17,22 @@ Your image does not need an SSH server preinstalled. CCluster injects one at dep

### Private registry images

To deploy an image from a private registry (e.g., Amazon ECR, Google Artifact Registry, Azure Container Registry, or a self-hosted registry), provide credentials in the **Image Registry Username** and **Image Registry Password** fields. CCluster automatically detects the registry from the image URL. For public images, leave both fields empty.
To deploy an image from a private registry (e.g., Amazon ECR, Google Artifact Registry, Azure Container Registry, or a self-hosted registry), set **Image pull authentication**. Choose **Username and password** and select **Image Registry Credentials** saved in [Secrets](/resources/vault), or choose **Existing cluster pull secret** and enter the **Cluster pull secret name** that your cluster administrator manages. CCluster automatically detects the registry from the image URL. For public images, leave authentication unset.

API deployments can instead [reference an existing image pull secret](/resources/custom-image#use-an-existing-image-pull-secret) managed by your cluster administrator using `image_pull_secret_name`.
API deployments can also [reference an existing image pull secret](/resources/custom-image#use-an-existing-image-pull-secret) using `image_pull_secret_name`.

## 2. Configure access and deploy

Enter your SSH public key to configure access to the instance, select a GPU instance type, and click Deploy.

Optional settings on the create form:

- **Storage** — mount filesystem volumes or S3 buckets on the **Hardware** step. Mount a writable volume at `/home` to keep your home files when the instance is paused. See [Managing Storage](/resources/storage).
- **Enable logging** — collect container output and show it on the deployment **Logs** tab.
- **Annotations** — optional key/value metadata on the instance.

<Frame>
<img src="./images/comp-1.png" alt="Compute instance launch page" />
<img src="./images/comp-1.png" alt="Compute Instance Hardware step with cluster, hardware cards, and storage" />
</Frame>

## 3. SSH into the instance
Expand Down Expand Up @@ -68,7 +69,7 @@ scp root@<endpoint_url>:/home/results.csv .
ssh -L 6006:localhost:6006 root@<endpoint_url>
```

X11 forwarding and SSH agent forwarding are not supported.
SSH agent forwarding is supported. X11 forwarding is not supported.

The interactive shell depends on what your image provides:

Expand All @@ -84,7 +85,7 @@ The instance includes the libraries from your image. Additional packages can be

While the instance is **Active**, the deployment **Terminal** tab opens a shell in the browser without SSH from your machine. Use SSH (above) for `scp`/`SFTP` and port forwarding. See [Managing Deployments](/resources/managing-deployments) for Terminal availability, pause/resume, clone, and logs.

Compute deployments are not revision-aware in the console: edits update the instance in place and there is no rollback table.
Compute deployments are revision-aware: spec edits create a new revision, and the **Detail** tab lists revisions so you can inspect them and roll back. See [Managing Deployments](/resources/managing-deployments#revisions-rollback-and-rollout).


## What's next
Expand All @@ -98,17 +99,17 @@ Compute deployments are not revision-aware in the console: edits update the inst
In-console Terminal, logs, pause, and clone.
</Card>
<Card
title="Managing Vault Objects"
title="Managing Secrets"
icon="fa-solid fa-key"
href="/resources/vault"
>
Store SSH keys and registry credentials for reuse across deployments
</Card>
<Card
title="LLM Serving"
icon="fa-solid fa-comments"
href="/apps/llm"
title="NVIDIA Dynamo"
icon="fa-solid fa-diagram-project"
href="/apps/dynamo"
>
Explore dedicated, protected endpoints for production model deployments.
Serve LLMs with disaggregated prefill/decode, autoscaling, and scale to zero.
</Card>
</CardGroup>
10 changes: 7 additions & 3 deletions docs/apps/dynamo.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ From **New Deployment**, choose **Dynamo**. The **Endpoint** step contains the r
- **Endpoint Name** — a unique name containing lowercase letters, numbers, and hyphens. It must begin with a lowercase letter, end with a letter or number, and be no more than 20 characters.
- **Model** — the Hugging Face model identifier loaded by the vLLM workers.
- **Served Model Name** — an optional alias clients can send in the OpenAI `model` field. If you leave it empty, use the Hugging Face model identifier.
- **Hugging Face Token** — required for gated and private models.
- **Hugging Face Token** — required for gated and private models. Select a token saved in [Secrets](/resources/vault).
- **Protocol** — `HTTP` for the OpenAI-compatible API or `gRPC` for the KServe v2 inference protocol.
- **Private Endpoint** — select or create a Bearer token or client certificate.
- **Annotations** — optional key/value metadata applied to the deployment components.
Expand Down Expand Up @@ -119,7 +119,7 @@ Prefill and Decode pools do not autoscale, and they cannot scale to zero.

### Storage volumes

Both topologies can mount reusable volumes from the cluster in the **Storage** section of the **Hardware** step. Mounts apply to the Frontend and to every worker's main container, at the same paths. Volumes must belong to the deployment's creator. When you edit a deployment, removing a mount changes only the new revision; the volume and its data are kept.
Both topologies can mount reusable volumes from the cluster in the **Storage** section of the **Hardware** step. To create volumes, see [Managing Storage](/resources/storage). Mounts apply to the Frontend and to every worker's main container, at the same paths. Volumes must belong to the deployment's creator. When you edit a deployment, removing a mount changes only the new revision; the volume and its data are kept.

Click **Deploy**. Model download and worker startup can take several minutes, especially for large models that are not already cached.

Expand Down Expand Up @@ -157,6 +157,10 @@ You can also create a disaggregated deployment through the API. Send a `POST` re

Each pool needs a `hardware_instance_id` and a fixed size: set `min_replicas` and `max_replicas` to the same value, at least `1`. The older per-pool `replicas` field is deprecated but still accepted. The Prefill and Decode pools can use the same GPU hardware profile or different profiles, but both profiles must be available in the selected cluster. Every replica consumes one complete hardware instance, so the deployment's GPU footprint is the sum of the Prefill and Decode pool footprints.

To pin the model to a Hugging Face branch, tag, or commit, set `model_revision`. NVIDIA CCluster resolves it to a commit and runs every worker at that commit.

On clusters that support GPU Direct RDMA, `"weight_transfer": {}` lets a starting worker copy the model weights GPU to GPU from a running worker of the same pool instead of downloading the checkpoint. It pins the model to a single commit, and it needs the same EFA-enabled AWS clusters as `require_gpu_direct_rdma`. When you enable it, `--revision` and `--load-format` in extra arguments are rejected, and environment variable names starting with `MX_` or `MODEL_EXPRESS_`, and `VLLM_PLUGINS`, are reserved.

The `endpoint_bearer_token` in this request protects the deployed inference endpoint. It is separate from the credential used to authenticate the request to the NVIDIA CCluster API.

`communications.kv_transfer.policy` selects the KV transfer policy: `allow_tcp`, the console default, or `require_gpu_direct_rdma`. If you omit `communications`, NVIDIA CCluster uses its default NIXL configuration, which also allows TCP. To pin a runtime version, set `runtime_version` to a Dynamo runtime version tag; otherwise NVIDIA CCluster selects the platform default for the cluster and policy.
Expand Down Expand Up @@ -244,7 +248,7 @@ curl "https://<endpoint_url>/v1/chat/completions" \

For a certificate-protected endpoint, configure your HTTP client with the downloaded client certificate instead of the `Authorization` header. See [Securing Endpoints](/resources/private).

If you selected `gRPC`, connect to `<endpoint_url>:443` with TLS and use Dynamo's KServe v2 `ModelInfer` or `ModelStreamInfer` API. This is different from the direct vLLM gRPC API used by General Inference deployments. See the [NVIDIA Dynamo KServe gRPC frontend guide](https://docs.nvidia.com/dynamo/latest/components/frontend/frontend-guide) for the protocol and [Securing Endpoints](/resources/private) for authentication.
If you selected `gRPC`, connect to `<endpoint_url>:443` with TLS and use Dynamo's KServe v2 `ModelInfer` or `ModelStreamInfer` API. This is different from the direct vLLM gRPC API used by General Inference deployments. See the [NVIDIA Dynamo KServe gRPC frontend guide](https://docs.nvidia.com/dynamo/components/frontend) for the protocol and [Securing Endpoints](/resources/private) for authentication.

## Current scope

Expand Down
Binary file modified docs/apps/images/comp-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/apps/images/dynamo-advanced-configuration.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/apps/images/dynamo-endpoint-configuration.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/apps/images/dynamo-hardware-scaling.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/apps/images/inf-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/apps/images/inf-2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/apps/images/inf-3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/apps/images/inf-4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/apps/images/llm-stage1.png
Binary file not shown.
Binary file removed docs/apps/images/llm-stage2.png
Binary file not shown.
32 changes: 16 additions & 16 deletions docs/apps/inference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,28 +7,28 @@ We make it easy for you to containerize and deploy any custom models on CCluster

## 1. Configure your inference deployment

Use the **Add or Select Image** dropdown to choose a pre-configured recipe or manually enter a custom image URL. Similarly, use the **Container Tag** dropdown to pick an available tag or type one in directly. If you select a recipe, all fields except **Add command** are auto-filled (some may be empty). You will still need to provide the entrypoint command under **Optional Details → Add command**.
Use the **Add or Select Image** dropdown to choose a pre-configured recipe or manually enter a custom image URL. Similarly, use the **Container Tag** dropdown to pick an available tag or type one in directly. If you select a recipe, all fields except **Add command** are auto-filled (some may be empty). You will still need to provide the entrypoint command in the **Advanced** step under **Add command**.

At a minimum, you need to specify:

- **Add or Select Image** — the container image URL (select from dropdown or enter manually).
- **Container Tag** — the image tag (select from dropdown or enter manually).
- **Container Port** — the port your container exposes its HTTP or gRPC service on.
- **Protocol** — select `HTTP` (default) or `gRPC`.
- **Health check path** — the endpoint used to verify readiness (e.g., `/health`, `/`). For gRPC deployments, CCluster uses TCP socket checks automatically, so this field is ignored.
- **Image Registry Username / Password** — credentials for private images (e.g., Docker Hub private repo). API deployments can instead [reference an existing image pull secret](/resources/custom-image#use-an-existing-image-pull-secret).
- **Image pull authentication** — for private images, choose **Username and password** and select **Image Registry Credentials** saved in [Secrets](/resources/vault), or choose **Existing cluster pull secret** and enter the **Cluster pull secret name** that your cluster administrator manages. Leave it unset for public images. See [Deploying Custom Models](/resources/custom-image#use-an-existing-image-pull-secret).
- **Hugging Face Token** — select a Hugging Face token saved in [Secrets](/resources/vault) for gated or private models. Do not pass `HF_TOKEN` as a plain environment variable.

The create wizard splits optional settings across **Advanced** and **Scaling**.
The create wizard has three steps: **Endpoint**, **Advanced**, and **Hardware**. Optional runtime settings are in **Advanced**, and **Scaling** settings are in **Hardware**.

**Advanced**

- **Enable logging** — collect container stdout/stderr and show it on the deployment **Logs** tab. If this is off, that revision has no logs in the console.
- **Enable Model Caching** — store model files in a node-local persistent cache. Restarts and replicas scheduled on the same node can reuse the files instead of downloading them again.
- **Session affinity** — best-effort sticky routing using the `X-Session-Id` request header. Requests with the same header value land on the same pod, which helps KV-cache reuse for agentic workloads. Requests without the header are routed at random. Affinity is **not** durable: scaling, rollouts, restarts, or readiness-probe transitions can remap sessions to different pods. Do not use it for irreplaceable in-pod state.
- **Health check path** — the endpoint used to verify readiness (also listed in the required fields above).
- **Health check path** — the endpoint used to verify readiness (e.g., `/health`, `/`). For gRPC deployments, CCluster uses TCP socket checks automatically, so this field is ignored.
- **Add command** — entrypoint command and arguments. If left empty, the image's default entrypoint is used.
- **Metrics port** / **Metrics path** — optional Prometheus scrape endpoint. Set **both** together, or leave both empty to use the defaults (the container port and `/metrics`).
- **Environment variables** — pass additional variables to the container (e.g., `HF_TOKEN`).
- **Environment variables** — pass additional variables to the container.
- **Config file** — mount a file into the container. Provide **Filename**, **Mount Path (directory)**, and **File Content**. For worked examples, see [gRPC Inference Endpoints](/resources/grpc) and [Deploying TensorRT-LLM](/resources/trtllm).
- **Annotations** — optional key/value metadata on the deployment.

Expand All @@ -40,19 +40,19 @@ CCluster scales replicas from in-flight request concurrency (**Max Concurrency**
| --- | --- |
| **Min Replicas** | Lower bound for replica count. `0` allows scale-to-zero when demand drops. |
| **Max Replicas** | Upper bound for replica count. Set equal to **Min Replicas** for a fixed size. |
| **Max Concurrency** | Target in-flight requests per replica. Required when **Max Replicas** is greater than **Min Replicas**. |
| **Max Concurrency** | Target in-flight requests per replica. Optional; leave unset for no concurrency target. |
| **Cooldown Period (s)** | Scale-down stabilization time, from `0` to `3600`. |

<Note>
Changing min or max replica counts on a running deployment scales in place and **does not create a new revision**. Spec changes such as image, command, or affinity do. See [Managing Deployments](/resources/managing-deployments).
</Note>

<Frame>
<img src="./images/inf-1.png" alt="General Inference deployment page" />
<img src="./images/inf-1.png" alt="General Inference Endpoint step with image, port, protocol, image pull authentication, Hugging Face token, and private endpoint fields" />
</Frame>

<Tip>
Every deployment must be protected — there are no public endpoints. Secure your endpoint with a Bearer token (an `Authorization: Bearer <token>` header), an mTLS client certificate, or both. To enable mTLS, select the "Make it a private endpoint?" option, which generates a TLS certificate you can download as a .pem file after the deployment is complete. For more details, see [Securing Endpoints](/resources/private).
Every deployment must be protected — there are no public endpoints. Secure your endpoint with a Bearer token (an `Authorization: Bearer <token>` header), an mTLS client certificate, or both. To enable mTLS, select the **Private Endpoint** option, which generates a TLS certificate you can download as a .pem file after the deployment is complete. For more details, see [Securing Endpoints](/resources/private).
</Tip>

<Tip>
Expand Down Expand Up @@ -112,10 +112,10 @@ For the full NIM configuration and an end-to-end walkthrough, see [Deploying NVI
By default, NVIDIA CCluster provides several managed clusters and GPU instances for you to deploy your inference containers.

<Frame>
<img src="./images/inf-2.png" alt="Hardware instance selection for general inference" />
<img src="./images/inf-2.png" alt="General Inference Hardware step with cluster, hardware cards, storage, and scaling settings" />
</Frame>

Select the regional cluster and hardware instance that best fits your need and click Deploy.
Select the regional cluster and hardware instance that best fits your need. To mount persistent storage, add a mount in the **Storage** section below the hardware cards; see [Managing Storage](/resources/storage). Then click Deploy.

<Tip>
You can integrate your own private cluster into CCluster through bring-your-own-infrastructure support. To get started, open a support request through your NVIDIA CCluster support channel.
Expand All @@ -126,7 +126,7 @@ You can integrate your own private cluster into CCluster through bring-your-own-
Once deployed, you can see all your deployments under the listing view along with their current status.

<Frame>
<img src="./images/inf-3.png" alt="Container image configuration for general inference" />
<img src="./images/inf-3.png" alt="Deployments list showing a general inference deployment and its status" />
</Frame>

Click on the deployment to view the details page, logs and monitoring information.
Expand Down Expand Up @@ -183,10 +183,10 @@ For gRPC deployments, see [gRPC Inference Endpoints](/resources/grpc) for connec
Learn how to protect your endpoints with Bearer tokens and mTLS certificates
</Card>
<Card
title="LLM Serving"
icon="fa-solid fa-comments"
href="/apps/llm"
title="NVIDIA Dynamo"
icon="fa-solid fa-diagram-project"
href="/apps/dynamo"
>
Explore dedicated, protected endpoints for production model deployments.
Serve LLMs with disaggregated prefill/decode, autoscaling, and scale to zero.
</Card>
</CardGroup>
Loading
Loading