ray-hive is a shared-cluster toolkit for Ray: it makes distributed Ray development simpler and more reliable by baking resilience into reusable primitives so each project ships less Ray glue and has a smoother path to production.
You run one heterogeneous Ray cluster (CPU workers for jobs, GPU workers for models). Multiple apps share that cluster — for example a quotes-pipeline (batch IO + Serve API), a catalog-search service (embeddings / retrieval), plus ad-hoc vLLM deploys and Train runs — without fighting over Serve routes, model ids, or cron windows. Hive owns the boring distributed concerns (placement envelopes, admission, durable queues, claim registry, Serve cool-down, VRAM-aware vLLM placement) so your project code stays domain-shaped.
Install extras match how you use the cluster:
pip install "ray-hive" # core: jobs + cluster + meta hooks
pip install "ray-hive[vllm]" # + Ray Serve / vLLM path
pip install "ray-hive[train]" # + Ray Train / custom infer
pip install "ray-hive[all]" # vllm + train
On connect you get one stdout line: ray-hive <ver> — https://github.com/BasicOverflow/ray-hive.
What you get
| Layer | Role |
|---|---|
vLLM Serve ([vllm]) |
Live-VRAM planning, heterogeneous GPU placement, sleep/idle, OpenAI HTTP |
| Jobs primitives (core) | IO/CPU envelopes, fan-out + admission, rate gates, durable queues, cron board |
Train ([train]) |
Custom PyTorch/HF loops with the same GPU claim model |
| ray-hive supervisor | Cluster-wide claims, cron calendar, sticky model revive (LXC/systemd host) |
Design intent: cut Ray-related code per project. Resilience (reconnect, cool-down, head/vram-monitor avoidance, durable pending work while Ray is down, collision GC) lives in hive so prod systems spend less time rediscovering the same failure modes.
- vLLM Serve
- Placement
- Allocation policies
- Custom attention
- Multimodal and context
- Embeddings
- Lifecycle
- OpenAI HTTP
- Jobs primitives
- ray-hive supervisor
- Train / custom infer
- Ops
- Tests
Ray Hive’s original job: run a heterogeneous Ray cluster and place vLLM model replicas according to live GPU memory. CPU workers keep running general Ray tasks; GPU workers host Serve replicas. KubeRay manages pods; hive adds planning, per-GPU placement, reservations, and routing.
Clients
└─ Ray head
├─ CPU workers → jobs / general Ray tasks
└─ GPU workers → vLLM replicas
├─ VRAM monitor DaemonSet
├─ shared GPU registry
└─ Ray Serve model routers
- Throughput-first planning — max practical
max_num_seqs/ batched tokens within a VRAM budget (gpu_budget_frac0.95). Withmax_input_prompt_length="auto"+ fixedmax_num_seqs, grow text context (floor 256) instead. Atmax_num_seqs=1, auto may shrink below 256 so weights + speculative draft still fit. Hybrid GDN/Mamba models cap auto input at 32768 so leftover VRAM is not spent on an unbounded attention window. - Heterogeneous replicas — per-GPU plans so different cards each contribute what they can.
- Least-loaded routing — relative to each replica’s planned capacity.
- Live VRAM scheduling — registry + reservations; see
examples/4_test_allocation_policies.py. - GPU sharing — co-locate when needed; intentional share via same
gpu=pin (examples/6_shared_gpu.py). - Flexible placement — pin, N replicas, or
replicas=-1(examples/1_test_model_configs.py). - Flagship deploy reference — Qwen3.8-27B + Nemotron 3.5 Lightning variants (
examples/15_flagship_models.py). - Same-node TP — auto escalate or pin a GPU list (
examples/5_tensor_parallel.py). - Custom attention — subclass using HF config fields (
examples/3_custom_attention.py). - Multimodal generate — image / video / audio (
examples/8–9,12–14). - Embeddings —
runner="pooling"(examples/10–11). - Sleep / idle — level-1 sleep then optional self-destroy (
examples/7_sleep_idle_timeout.py). - OpenAI HTTP — per-model routes + cluster
/v1gateway (examples/2_test_inference.py).
Out of scope: precomputed multimodal embeds as input (enable_mm_embeds).
Dry-run packing against live GPUs (same plan deploy would use):
from ray_hive import RayHive
hive = RayHive(address="ray://YOUR_RAY_HEAD_IP:10001")
plan = hive.estimate_vram(
"Qwen/Qwen3-0.6B-FP8",
max_input_prompt_length=1024,
max_output_prompt_length=2048,
replicas=1,
vllm_kwargs={
"trust_remote_code": True,
"reasoning_parser": "qwen3",
"default_chat_template_kwargs": {"enable_thinking": False},
},
)Deployment Plan: Qwen/Qwen3-0.6B-FP8
Replica replica-0
GPU(s) host-a:gpu0
tensor_parallel_size 1
max_input_prompt_length 1024
max_output_prompt_length 2048
max_model_len 3072
max_num_seqs 48
max_num_batched_tokens 8192
gpu_memory_utilization 0.850
Weights 0.60 GiB
KV cache 4.20 GiB
Activations 0.40 GiB
Overhead 0.30 GiB
Total (per GPU) 5.50 GiB
Deploy until ready (router up + warmed):
status = hive.deploy_model(
model_id="qwen",
model_name="Qwen/Qwen3-0.6B-FP8",
max_input_prompt_length=1024,
max_output_prompt_length=2048,
replicas=1,
vllm_kwargs={
"trust_remote_code": True,
"reasoning_parser": "qwen3",
"default_chat_template_kwargs": {"enable_thinking": False},
},
){"model_id": "qwen", "status": "ready", "route": "/qwen", "openai_v1": "/v1", "replicas": {...}}
Generate text (batch and stream helpers available):
from ray_hive.inference import inference, inference_batch, inference_stream
answer = inference("Explain Ray in one sentence.", model_id="qwen")
answers = inference_batch(["Prompt one", "Prompt two"], model_id="qwen")
for delta in inference_stream("Count to three.", model_id="qwen"):
print(delta, end="")Structured output:
from pydantic import BaseModel
class Answer(BaseModel):
summary: str
confidence: float
result = inference("Summarize Ray.", model_id="qwen", structured_output=Answer)Inside an asyncio loop use a_inference / a_inference_batch instead of the sync helpers.
gpu="host:gpu0"— one replica on one GPU.gpu=[a,b,...]+replicas=len(list)— N single-GPU pins (TP=1 each).gpu=[a,b,...]+replicas=1— one same-node TP group.gpu=None— auto place (single GPU viaallocation_cls, defaultRayPerformanceAllocator; else same-node TP).replicas=-1— every eligible GPU / TP group.
By default context lengths are a fixed contract and omitted max_num_seqs is packed to fill VRAM.
max_input_prompt_length="auto" does the inverse for text input only: you must pass max_num_seqs in vllm_kwargs, output length stays fixed, and the planner grows text input from a floor of 256 until the VRAM budget is filled (capped by HF max_position_embeddings / model_max_length when present). When max_num_seqs=1 and even 256 does not leave KV room after weights + speculative draft, auto shrinks toward 1 token. Multimodal placeholder tokens are still added on top of the chosen text length, so max_model_len always covers MM + output.
speculative_config is part of the memory plan: in-checkpoint MTP reserves an extra compute-dtype lm_head (and num_nextn_predict_layers when set); a separate draft model adds that checkpoint’s weights and KV.
hive.estimate_vram(
"Qwen/Qwen3-0.6B-FP8",
max_input_prompt_length="auto",
max_output_prompt_length=512,
replicas=1,
vllm_kwargs={
"max_num_seqs": 8,
"trust_remote_code": True,
"reasoning_parser": "qwen3",
"default_chat_template_kwargs": {"enable_thinking": False},
},
)Passing "auto" without max_num_seqs raises ConfigError.
Pin one GPU:
hive.deploy_model(
model_id="qwen-pin",
model_name="Qwen/Qwen3-0.6B-FP8",
max_input_prompt_length=512,
max_output_prompt_length=512,
replicas=1,
gpu="ergos-06-nv:gpu0",
vllm_kwargs={"trust_remote_code": True, "reasoning_parser": "qwen3",
"default_chat_template_kwargs": {"enable_thinking": False}},
)All eligible GPUs: replicas=-1. Same-node TP=2: pass a GPU list with replicas=1. Live registry: hive.get_vram_state().
When gpu= is unset, allocation_cls picks GPUs (ray_hive.core.ray_gpu_alloc). Pins skip policy ranking but still check VRAM fit and arch taints. Default: RayPerformanceAllocator.
RayPerformanceAllocator— rank by compute (SM count, light bandwidth tie-break); top-N.RayConserveTdpAllocator— prefer lower approx TDP; SM count tie-break.RayTensorParallelAllocator— same-node packs of sizetensor_parallel_size; auto when single-GPU fails.
Arch taint: native FP8 needs compute capability ≥ 8.9 (Ada+); Ampere dropped automatically.
TP=1 policies prefer unshared GPUs first, then co-locate. Alive Ray nodes only. See examples/4_test_allocation_policies.py and examples/5_tensor_parallel.py.
VRAM planning sizes the KV cache through an attention-specs class. Default BaseAttentionSpecs (or MultimodalAttentionSpecs for MM). Pass a subclass via attention_cls on estimate_vram / deploy_model.
from ray_hive.core.model_specs import BaseAttentionSpecs
class MyAttention(BaseAttentionSpecs):
@property
def kv_heads(self) -> int:
return self.hf_params["num_key_value_heads"]
hive.deploy_model(..., attention_cls=MyAttention)See examples/3_custom_attention.py.
Text-only models — pass a string (or text chat messages). No limit_mm_per_prompt.
MM models — planner uses limit_mm_per_prompt in vllm_kwargs to size worst-case image / video / audio placeholders. If omitted on an MM HF config, defaults come from vision_config / audio_config. Set counts explicitly to enable or disable modalities.
Token budget — max_input_prompt_length is the text side only. Effective input ≈ text + MM placeholders; max_model_len ≈ effective_input + max_output_prompt_length (use output 0 for pooling). If that cannot cover placeholders + output, planning raises MmContextError. With "auto" input, text grows from 256 while MM placeholders stay reserved.
Requests vs planning — on an MM deploy you can still send text-only strings; that does not shrink the VRAM plan. For text-only planning on an MM checkpoint, zero unused modalities ({"image": 0, "video": 0, "audio": 0}).
from ray_hive.core.ray_utils import file_to_data_url
from ray_hive.inference import inference
hive.deploy_model(
model_id="vl",
model_name="Qwen/Qwen2.5-VL-3B-Instruct",
max_input_prompt_length=2048,
max_output_prompt_length=256,
replicas=1,
vllm_kwargs={"trust_remote_code": True, "limit_mm_per_prompt": {"image": 1}},
)
messages = [{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": file_to_data_url("photo.png")}},
{"type": "text", "text": "Describe this image."},
],
}]
answer = inference(messages, model_id="vl", max_tokens=64)Content part types: text, image_url, video_url / video, audio_url / input_audio. Fixtures: examples/media/. Demos: examples/8–9, 12–14.
Set runner="pooling" (legacy task="embed" still works). Use max_output_prompt_length=0. inference returns a vector (or list of vectors for batch).
hive.deploy_model(
model_id="embed",
model_name="BAAI/bge-small-en-v1.5",
max_input_prompt_length=512,
max_output_prompt_length=0,
replicas=1,
vllm_kwargs={"runner": "pooling", "trust_remote_code": True},
)
vec = inference("hello", model_id="embed")curl http://YOUR_RAY_HEAD_IP:8000/embed/v1/embeddings \
-H "Content-Type: application/json" \
-d '{"model":"embed","input":["hello world"]}'See examples/10_text_embeddings.py and examples/11_multimodal_embeddings.py.
Sleep after quiet, then optional full destroy (idle_timeout must be greater than sleep_timeout when both are set). While sleeping, the running footprint stays reserved so other deploys cannot take the space a wake needs. Pass sleep_peak_factor in vllm_kwargs only if you need an extra weight-sized hold.
hive.deploy_model(
model_id="qwen",
model_name="Qwen/Qwen3-0.6B-FP8",
max_input_prompt_length=512,
max_output_prompt_length=256,
replicas=1,
sleep_timeout=60,
idle_timeout=300,
vllm_kwargs={
"trust_remote_code": True,
"reasoning_parser": "qwen3",
"default_chat_template_kwargs": {"enable_thinking": False},
},
)
hive.shutdown("qwen") # one model
hive.shutdown() # all *hive* models only (never serve.shutdown — other apps stay up)
free = hive.free_gpus(min_free_gb=8.0)from ray_hive import kill_gpu_registry
kill_gpu_registry() # explicit rebuild only — not part of normal shutdownSee examples/7_sleep_idle_timeout.py and examples/0_shutdown_models.py. Sticky pin when HIVE_SUPERVISOR_STATE is set (ray-hive supervisor). GCS FT Redis: manifests/ray-gcs-redis.yaml.
Per-model route and cluster-wide /v1 gateway (port 8000). Gateway starts on deploy_model; for an already-running cluster call hive.ensure_openai_api().
curl http://YOUR_RAY_HEAD_IP:8000/qwen/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"qwen","messages":[{"role":"user","content":"Explain Ray briefly."}]}'
# Open WebUI: API URL = http://YOUR_RAY_HEAD_IP:8000/v1
curl http://YOUR_RAY_HEAD_IP:8000/v1/modelsStreaming, structured JSON, and LangChain: examples/2_test_inference.py.
These are the building blocks for non-LLM Ray work (and for the CPU side of LLM apps). They exist so every project does not re-implement placement, backpressure, and crash recovery.
What: a named claim book for the whole cluster.
Why: two projects must not both own /quotes, the same model_id, or the same cron id.
Where: detached actor in namespace ray_hive_system.
from ray_hive.cluster import ProjectContext, init_ray_for_project
ctx = ProjectContext(
project_id="quotes-pipeline",
namespace="quotes-pipeline",
serve_route_prefix="/quotes",
)
init_ray_for_project("ray://YOUR_RAY_HEAD_IP:10001", ctx)Miss heartbeats long enough and the ray-hive supervisor GC marks the project dead and releases claims.
What: @ray.remote option packs with safe defaults.
Why: accidental scheduling onto the Ray head or onto vram-monitor pods (memory-only, no CPU) silently fails or starves the head.
Rule: workload envelopes always set label_selector so tasks stay on worker node-groups (!headgroup). No opt-out. Use ops_remote for head / every-node ops (Serve restart, disk prune).
from ray_hive.jobs import io_remote, cpu_remote
@io_remote()
def fetch_chunk(urls: list[str]) -> list:
...
@cpu_remote(num_cpus=-1) # -1 → AdmissionBroker suggests width
def stitch(batch: list) -> list:
...What: submit work in chunks with ray.wait backpressure; width comes from live cluster CPU/RAM, not a hardcoded MAX_TASKS.
Why: fixed concurrency either underuses the cluster or OOMs it when another project is busy.
from ray_hive.jobs import BatchFanout, TaskShape
fan = BatchFanout()
shape = TaskShape(batch_size=32, max_ceiling=64, cpus_per_task=0.001)
fan.run("ingest", items, lambda batch: fetch_chunk.remote(batch), shape)max_ceiling / max_in_flight are safety rails. If the broker actor is down, fan-out still runs with those rails.
What: a named cluster actor that serializes / paces external HTTP or vendor APIs.
Why: per-process sleeps do not coordinate across workers.
from ray_hive.jobs import get_or_create_rate_gate
import ray
gate = get_or_create_rate_gate(
"vendor-http-gate",
namespace="quotes-pipeline",
min_interval=2.5,
)
ray.get(gate.acquire.remote())
# call external API
ray.get(gate.record_success.remote())What: SQLite-backed pending/running/done queue on the project supervisor host.
Why: Ray (and the whole worker set) can be offline for hours; catch-up work must survive.
from ray_hive.jobs import DurableJobQueue
q = DurableJobQueue("/var/lib/quotes-pipeline/queue.db")
q.enqueue("daily-bar-sync", {"day": "2024-06-01"})
job = q.claim_next(["daily-bar-sync"])
q.complete(job["id"])Stale running rows requeue after stale_running_s.
What: project declares schedules; meta holds a cluster-wide board (next_fire, load_class).
Why: utilization planning across projects (IO vs CPU vs GPU windows) without one supervisor owning everyone’s wall clock.
from ray_hive.jobs import CronSchedule, get_cron_board
import ray
CronSchedule(
"quotes-daily-bar-sync",
"quotes-pipeline",
"16:05",
load_class="io",
)
board = get_cron_board()
print(ray.get(board.list_entries.remote()))Each project’s supervisor still fires the work; the board is the shared calendar.
What: durable “which ids are done” for long fan-outs (embeddings, backfills).
Why: restart mid-job without redoing completed shards.
from ray_hive.jobs import JobProgress
prog = JobProgress("/var/lib/catalog-search/progress.db", "embed-v1")
todo = prog.pending(all_ids)
for i in todo:
...
prog.mark_done(i)- ScratchShardStore — local scratch generations under a prefix; safe wipe of orphan trees.
- ServeAppHeal — deploy/replace a Serve app with cool-down + route claim (so two projects do not stomp
/quotes). - NodeBroadcast — one ops task per alive node (disk prune, cache flush).
- MapShuffleReduce — small map→shuffle→reduce helper for CPU pipelines.
Request coalescing (identical in-flight HTTP→remote) stays in your app, not in hive.
LXC/systemd entry: python -m ray_hive.supervisor.main (deploy/; unit ray-hive-supervisor).
Keeps ClusterRegistry, AdmissionBroker, and cron board warm, GCs dead projects, and redeploys sticky vLLM models when claimed. It does not replace per-project supervisors (e.g. quotes-pipeline or catalog-search) — those register into it and own their domain queues / Serve apps.
State file: HIVE_SUPERVISOR_STATE (SQLite): sticky models, projects, cron_board mirror, queues, trains, claims.
bash deploy/install.sh
# journalctl -u ray-hive-supervisor -f # on the supervisor hostWorker images must include ray-hive (CPU: manifests/Dockerfile.ray-cpu-hive; GPU: Dockerfile.ray-vllm-worker) so system actors created via Ray Client can import the package on the cluster.
Sibling to vLLM — your nn.Module / HF code, VRAM claims via the same registry.
from ray_hive.train import TrainJob, gpu_remote
@gpu_remote()
def worker_step(batch):
...
job = TrainJob("run-1", "gpu-rack-a", train_fn=my_loop, num_workers=2)
job.run()CustomInferService claims a model_id then calls your deploy/shutdown callables.
GCS FT (Redis): manifests/ray-gcs-redis.yaml + gcsFaultToleranceOptions on the RayCluster. After head failover, Serve controllers can revive on surviving workers. Hive’s in-memory gpu_registry does not survive GCS alone — the supervisor recreates it.
Redis is pinned to dedicated Longhorn nodes (longhorn-node=true) with 30s NotReady eviction so the RWO PVC can reattach when a Proxmox host dies. Apply longhorn-node-down-policy.yaml and ray-dead-node-janitor.yaml with it: Longhorn must delete volume-holding pods on down nodes, and the janitor force-deletes Terminating Ray pods / stale VolumeAttachments that kubelet can no longer acknowledge.
kubectl apply -f manifests/longhorn-node-down-policy.yaml
kubectl apply -f manifests/ray-gcs-redis.yaml
kubectl apply -f manifests/ray-dead-node-janitor.yaml
kubectl apply -f manifests/kuberay-operator.yaml
kubectl apply -f manifests/raycluster.yamlImages
| Role | Dockerfile | Example tag |
|---|---|---|
| Head + CPU workers | manifests/Dockerfile.ray-cpu-hive |
…/ray-cpu-hive:2.56.0-0.2.1 |
| GPU workers | manifests/Dockerfile.ray-vllm-worker |
…/ray-vllm-worker:2.56.0-vllm028-ocr2-0.2.1 |
Gitea PyPI (LAN): http://10.0.0.52:3000/api/packages/admin/pypi/simple/ — used by CT installs and as a runtime_env fallback when creating system actors.
Repository layout
ray_hive/— vLLM planner/registry/deploy, jobs, cluster, train, supervisordeploy/— LXC/systemd install for the ray-hive supervisormanifests/— KubeRay cluster, GCS Redis FT, worker images, VRAM monitorexamples/— deploy/inference experimentstests/— unit / Ray smoke / live / chaos
Related: rayify (script → Ray job packaging).
pip install -e ".[dev]"
# Parallel unit suite (no Ray / no LLM)
pytest -n auto -m "not ray and not live"ray / live require RAY_ADDRESS:
export RAY_ADDRESS=ray://YOUR_RAY_HEAD_IP:10001 # or PowerShell $env:RAY_ADDRESS=...
pytest -m ray # cluster smoke, no LLM
pytest -m live # Deploy cycles; needs GPUs + model downloadDo not use pytest -n for live GPU claims. Markers: ray, live, nightly, chaos.