Repository navigation
Add Beam 1.0 NVFP4 recipe for G4 with SGLang, DFlash2 and HiCache - #288
Conversation
Serves reflection/Beam-1.0-501B-A23B-NVFP4 on one g4-standard-384 node with the official lmsysorg/sglang:beam image: TP8/EP8, DFlash2 speculative decoding, and HiCache host-memory KV cache. Weights download from Hugging Face at launch into a pod-scoped volume. - inference/g4/beam_501b/single-host-serving/sglang: README, Helm values, SM120 MoE runner patch, AgentX benchmark job - src/helm-charts/g4/inference-templates/sglang-deployment: G4 SGLang chart - src/frameworks/g4/sglang-configs/beam_nvfp4_hicache_dflash.yaml: SGLang server arguments
There was a problem hiding this comment.
Code Review
This pull request introduces a recipe and Helm chart to serve the Reflection Beam 1.0 501B NVFP4 model on a single 8-GPU G4 node in GKE using SGLang, DFlash2, and HiCache, along with an AgentX benchmark client. The review feedback focuses on improving the robustness of the benchmark client script (bench_client.yaml) by enabling shell fail-on-error (set -e), explicitly validating that the server is ready before starting, handling masked exit statuses in piped commands, and ensuring the Python reporting script handles missing files and uses proper context managers.
| - bash | ||
| - -c | ||
| - | | ||
| set -u |
| for i in $(seq 1 540); do | ||
| curl -sf "http://$TARGET_HOST:$TARGET_PORT/ready" >/dev/null && break | ||
| sleep 10 | ||
| done |
There was a problem hiding this comment.
If the server fails to start or doesn't become ready within the 90-minute timeout, the script currently continues to run the benchmark anyway, which will fail with obscure errors and waste cluster resources. Adding an explicit check and exiting with an error status prevents this.
SERVER_READY=0
for i in $(seq 1 540); do
if curl -sf "http://$TARGET_HOST:$TARGET_PORT/ready" >/dev/null; then
SERVER_READY=1
break
fi
sleep 10
done
if [ $SERVER_READY -ne 1 ]; then
echo "Error: Server did not become ready within 90 minutes." >&2
exit 1
fi| agentx profile \ | ||
| --scenario inferencex-agentx-mvp \ | ||
| --url "http://$TARGET_HOST:$TARGET_PORT" \ | ||
| --endpoint-type chat --streaming \ | ||
| --model "$SERVED_MODEL" \ | ||
| --tokenizer "$TOKENIZER" --tokenizer-trust-remote-code \ | ||
| --public-dataset "$DATASET" \ | ||
| --max-context-length "$MAX_CONTEXT" \ | ||
| --concurrency "$C" \ | ||
| --benchmark-duration "$BENCH_DURATION" \ | ||
| --use-server-token-count \ | ||
| --extra-inputs ignore_eos:true \ | ||
| --cache-bust first_turn_prefix \ | ||
| --system-idle-gap-cap-seconds 10 \ | ||
| --trajectory-start-min-ratio 0.0 --trajectory-start-max-ratio 1.0 \ | ||
| --random-seed "$SEED" \ | ||
| --ui simple $EXTRA \ | ||
| --artifact-dir "/results/artifacts-c${C}" 2>&1 | tee "/results/agentx-c${C}.log" | grep -vE "^\s*$" |
There was a problem hiding this comment.
Since the agentx profile command is piped to tee and grep, its exit status is masked by default (even with set -e). Checking ${PIPESTATUS[0]} ensures that if the benchmark tool fails, the script immediately exits with an error instead of silently continuing to the next concurrency level.
agentx profile \
--scenario inferencex-agentx-mvp \
--url "http://$TARGET_HOST:$TARGET_PORT" \
--endpoint-type chat --streaming \
--model "$SERVED_MODEL" \
--tokenizer "$TOKENIZER" --tokenizer-trust-remote-code \
--public-dataset "$DATASET" \
--max-context-length "$MAX_CONTEXT" \
--concurrency "$C" \
--benchmark-duration "$BENCH_DURATION" \
--use-server-token-count \
--extra-inputs ignore_eos:true \
--cache-bust first_turn_prefix \
--system-idle-gap-cap-seconds 10 \
--trajectory-start-min-ratio 0.0 --trajectory-start-max-ratio 1.0 \
--random-seed "$SEED" \
--ui simple $EXTRA \
--artifact-dir "/results/artifacts-c${C}" 2>&1 | tee "/results/agentx-c${C}.log" | grep -vE "^\s*$"
if [ ${PIPESTATUS[0]} -ne 0 ]; then
echo "Error: AgentX benchmark failed for concurrency $C" >&2
exit 1
fi| python3 - <<'EOF' | ||
| import glob, json | ||
| for p in sorted(glob.glob("/results/artifacts-c*/**/profile_export_aiperf.json", recursive=True)): | ||
| r = json.load(open(p)) | ||
| md = r.get("metadata", {}) | ||
| print(f"### AGENTX_RESULT {p}") | ||
| print("submission_valid:", md.get("submission_valid"), md.get("submission_invalid_reasons")) | ||
| for k in ("request_throughput", "output_token_throughput", "time_to_first_token", | ||
| "inter_token_latency", "request_latency", "request_count", "error_request_count", | ||
| "context_overflow_count", "input_sequence_length", "output_sequence_length"): | ||
| v = r.get(k) | ||
| if isinstance(v, dict): | ||
| print(f"{k:28s} avg={v.get('avg')} p50={v.get('p50')} p90={v.get('p90')} p99={v.get('p99')} unit={v.get('unit')}") | ||
| elif v is not None: | ||
| print(f"{k:28s} {v}") | ||
| EOF |
There was a problem hiding this comment.
If no benchmark profile export files are found (e.g., if the benchmark failed to run or output results), the python script currently exits silently with status 0. Adding a check to ensure results were found and using standard Python file context managers (with open) makes the reporting step much more robust.
python3 - <<'EOF'
import glob, json, sys
results_found = False
for p in sorted(glob.glob("/results/artifacts-c*/**/profile_export_aiperf.json", recursive=True)):
results_found = True
with open(p, 'r', encoding='utf-8') as f:
r = json.load(f)
md = r.get("metadata", {})
print(f"### AGENTX_RESULT {p}")
print("submission_valid:", md.get("submission_valid"), md.get("submission_invalid_reasons"))
for k in ("request_throughput", "output_token_throughput", "time_to_first_token",
"inter_token_latency", "request_latency", "request_count", "error_request_count",
"context_overflow_count", "input_sequence_length", "output_sequence_length"):
v = r.get(k)
if isinstance(v, dict):
print(f"{k:28s} avg={v.get('avg')} p50={v.get('p50')} p90={v.get('p90')} p99={v.get('p99')} unit={v.get('unit')}")
elif v is not None:
print(f"{k:28s} {v}")
if not results_found:
print("Error: No benchmark profile export files found!", file=sys.stderr)
sys.exit(1)
EOF
Summary
Adds a single-node GKE recipe that serves Reflection's Beam 1.0 NVFP4 (
reflection/Beam-1.0-501B-A23B-NVFP4) on oneg4-standard-384node (8× RTX PRO 6000 Blackwell Server Edition). It uses the officiallmsysorg/sglang:beamimage, DFlash2 speculative decoding (reflection/Beam-1.0-501B-A23B-DFlash2), and HiCache, a pinned host-memory KV cache tier. It complements the G4 llm-d + vLLM Beam recipe added in #286.Serves reflection/Beam-1.0-501B-A23B-NVFP4 on one g4-standard-384 node with the official lmsysorg/sglang:beam image: TP8/EP8, DFlash2 speculative decoding, and HiCache host-memory KV cache. Weights download from Hugging Face at launch into a pod-scoped volume.
What's included
inference/g4/beam_501b/single-host-serving/sglang/sm120-beam-moe.patch, AgentX benchmark Jobsrc/helm-charts/g4/inference-templates/sglang-deployment/--set-file serving_config=...src/frameworks/g4/sglang-configs/beam_nvfp4_hicache_dflash.yamlflashinfer_cutlassMoE,trtllm_mha, DFlash2 (block size 8), HiCache (ratio 2.0,write_back)Notes
f7473673dd), Beam allows only theflashinfer_trtllmMoE runner for NVFP4. A one-line patch, applied at container start, also allowsflashinfer_cutlass, which is the runner used on G4 (SM120).dynamic-rwoStorageClass.Testing
On GKE
1.35.8-gke.1439001, oneg4-standard-384node:helm lint,helm template, andkubectl apply --dry-run=serverpass.Smoke test through the Service:
/v1/modelsreturnsbeam, and a chat completion returns a coherent answer with thebeamreasoning parser.Fixed ISL/OSL sweep (1k/1k, 8k/1k, 1k/8k at concurrency 32, 64, and 128): all 9 runs completed with no failed requests. These numbers are not part of this PR.
Code Changes hightlight