Kueue integration for LLMKube: an
external jobframework
component that lets Kueue admit, queue, and
account GPU quota for InferenceService workloads, so one stack manages GPU
budgets for both training and inference.
- An
InferenceServicelabeled withkueue.x-k8s.io/queue-namestarts suspended (a mutating webhook defaultsspec.suspend: true). - This component implements Kueue's
GenericJobinterface forInferenceServiceand creates the corresponding KueueWorkload. - When the ClusterQueue admits it, the service is unsuspended and starts.
- Scaling to zero deactivates the Workload and releases its quota; scaling back up re-queues for admission.
- Unlabeled InferenceServices are untouched: LLMKube's built-in
GPUQuotakeeps gating them, and it defers to Kueue for queue-labeled services.
The recommended topology keeps inference in a high-priority ClusterQueue that
lends idle GPU (lendingLimit) while batch borrows and is the preemptible
side. Live inference is not preempted.
Modeled on konflux-ci/tekton-kueue, per the guidance in LLMKube#1249.
Early development. Tracking:
epic LLMKube#1253,
component issue LLMKube#1252,
and this repo's issues. See docs/install.md for prerequisites and
deployment steps.
- docs/install.md: prerequisites, deploying the controller and webhook, registering the external framework with Kueue, and failure modes.
- docs/queue-topology.md: a walkthrough of the
sample lend/borrow queue topology in
config/samples/kueue-topology.yaml.
Apache-2.0