Skip to content

Add Beam 1.0 NVFP4 recipe for G4 with SGLang, DFlash2 and HiCache - #288

Merged
Chris113113 merged 1 commit into
AI-Hypercomputer:mainfrom
shivajid:g4-beam-nvfp4-sglang-hicache
Oct 19, 2026
Merged

Chris113113 merged 1 commit into
AI-Hypercomputer:mainfrom
shivajid:g4-beam-nvfp4-sglang-hicache

Conversation

@shivajid

Copy link
Copy Markdown
Contributor

Summary

Adds a single-node GKE recipe that serves Reflection's Beam 1.0 NVFP4 (reflection/Beam-1.0-501B-A23B-NVFP4) on one g4-standard-384 node (8× RTX PRO 6000 Blackwell Server Edition). It uses the official lmsysorg/sglang:beam image, DFlash2 speculative decoding (reflection/Beam-1.0-501B-A23B-DFlash2), and HiCache, a pinned host-memory KV cache tier. It complements the G4 llm-d + vLLM Beam recipe added in #286.

Serves reflection/Beam-1.0-501B-A23B-NVFP4 on one g4-standard-384 node with the official lmsysorg/sglang:beam image: TP8/EP8, DFlash2 speculative decoding, and HiCache host-memory KV cache. Weights download from Hugging Face at launch into a pod-scoped volume.

What's included

Path Purpose
inference/g4/beam_501b/single-host-serving/sglang/ README, Helm values, sm120-beam-moe.patch, AgentX benchmark Job
src/helm-charts/g4/inference-templates/sglang-deployment/ Helm chart: Deployment, Service, and patch ConfigMap. SGLang flags come from a framework config passed with --set-file serving_config=...
src/frameworks/g4/sglang-configs/beam_nvfp4_hicache_dflash.yaml SGLang server arguments: TP8/EP8, flashinfer_cutlass MoE, trtllm_mha, DFlash2 (block size 8), HiCache (ratio 2.0, write_back)

Notes

  • SM120 patch: in the image's SGLang (f7473673dd), Beam allows only the flashinfer_trtllm MoE runner for NVFP4. A one-line patch, applied at container start, also allows flashinfer_cutlass, which is the runner used on G4 (SM120).
  • Pinned for repeatability: the image by digest, and the model and drafter by Hugging Face revision.
  • No pre-staged disk: the weights (~298 GB) download at launch into a 500Gi pod-scoped volume from the dynamic-rwo StorageClass.

Testing

On GKE 1.35.8-gke.1439001, one g4-standard-384 node:

  • helm lint, helm template, and kubectl apply --dry-run=server pass.

  • Smoke test through the Service: /v1/models returns beam, and a chat completion returns a coherent answer with the beam reasoning parser.

  • Fixed ISL/OSL sweep (1k/1k, 8k/1k, 1k/8k at concurrency 32, 64, and 128): all 9 runs completed with no failed requests. These numbers are not part of this PR.

Code Changes hightlight

  • inference/g4/beam_501b/single-host-serving/sglang: README, Helm values, SM120 MoE runner patch, AgentX benchmark job
  • src/helm-charts/g4/inference-templates/sglang-deployment: G4 SGLang chart
  • src/frameworks/g4/sglang-configs/beam_nvfp4_hicache_dflash.yaml: SGLang server arguments

Serves reflection/Beam-1.0-501B-A23B-NVFP4 on one g4-standard-384 node with
the official lmsysorg/sglang:beam image: TP8/EP8, DFlash2 speculative
decoding, and HiCache host-memory KV cache. Weights download from Hugging
Face at launch into a pod-scoped volume.
- inference/g4/beam_501b/single-host-serving/sglang: README, Helm values,
  SM120 MoE runner patch, AgentX benchmark job
- src/helm-charts/g4/inference-templates/sglang-deployment: G4 SGLang chart
- src/frameworks/g4/sglang-configs/beam_nvfp4_hicache_dflash.yaml: SGLang
  server arguments

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a recipe and Helm chart to serve the Reflection Beam 1.0 501B NVFP4 model on a single 8-GPU G4 node in GKE using SGLang, DFlash2, and HiCache, along with an AgentX benchmark client. The review feedback focuses on improving the robustness of the benchmark client script (bench_client.yaml) by enabling shell fail-on-error (set -e), explicitly validating that the server is ready before starting, handling masked exit statuses in piped commands, and ensuring the Python reporting script handles missing files and uses proper context managers.

- bash
- -c
- |
set -u

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Enable shell fail-on-error (set -e) to ensure that any setup failures (such as apt-get or pip install failing) immediately terminate the benchmark job instead of silently continuing with an invalid environment.

          set -eu

Comment on lines +98 to +101
for i in $(seq 1 540); do
curl -sf "http://$TARGET_HOST:$TARGET_PORT/ready" >/dev/null && break
sleep 10
done

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If the server fails to start or doesn't become ready within the 90-minute timeout, the script currently continues to run the benchmark anyway, which will fail with obscure errors and waste cluster resources. Adding an explicit check and exiting with an error status prevents this.

          SERVER_READY=0
          for i in $(seq 1 540); do
            if curl -sf "http://$TARGET_HOST:$TARGET_PORT/ready" >/dev/null; then
              SERVER_READY=1
              break
            fi
            sleep 10
          done
          if [ $SERVER_READY -ne 1 ]; then
            echo "Error: Server did not become ready within 90 minutes." >&2
            exit 1
          fi

Comment on lines +109 to +126
agentx profile \
--scenario inferencex-agentx-mvp \
--url "http://$TARGET_HOST:$TARGET_PORT" \
--endpoint-type chat --streaming \
--model "$SERVED_MODEL" \
--tokenizer "$TOKENIZER" --tokenizer-trust-remote-code \
--public-dataset "$DATASET" \
--max-context-length "$MAX_CONTEXT" \
--concurrency "$C" \
--benchmark-duration "$BENCH_DURATION" \
--use-server-token-count \
--extra-inputs ignore_eos:true \
--cache-bust first_turn_prefix \
--system-idle-gap-cap-seconds 10 \
--trajectory-start-min-ratio 0.0 --trajectory-start-max-ratio 1.0 \
--random-seed "$SEED" \
--ui simple $EXTRA \
--artifact-dir "/results/artifacts-c${C}" 2>&1 | tee "/results/agentx-c${C}.log" | grep -vE "^\s*$"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Since the agentx profile command is piped to tee and grep, its exit status is masked by default (even with set -e). Checking ${PIPESTATUS[0]} ensures that if the benchmark tool fails, the script immediately exits with an error instead of silently continuing to the next concurrency level.

            agentx profile \
              --scenario inferencex-agentx-mvp \
              --url "http://$TARGET_HOST:$TARGET_PORT" \
              --endpoint-type chat --streaming \
              --model "$SERVED_MODEL" \
              --tokenizer "$TOKENIZER" --tokenizer-trust-remote-code \
              --public-dataset "$DATASET" \
              --max-context-length "$MAX_CONTEXT" \
              --concurrency "$C" \
              --benchmark-duration "$BENCH_DURATION" \
              --use-server-token-count \
              --extra-inputs ignore_eos:true \
              --cache-bust first_turn_prefix \
              --system-idle-gap-cap-seconds 10 \
              --trajectory-start-min-ratio 0.0 --trajectory-start-max-ratio 1.0 \
              --random-seed "$SEED" \
              --ui simple $EXTRA \
              --artifact-dir "/results/artifacts-c${C}" 2>&1 | tee "/results/agentx-c${C}.log" | grep -vE "^\s*$"
            if [ ${PIPESTATUS[0]} -ne 0 ]; then
              echo "Error: AgentX benchmark failed for concurrency $C" >&2
              exit 1
            fi

Comment on lines +129 to +144
python3 - <<'EOF'
import glob, json
for p in sorted(glob.glob("/results/artifacts-c*/**/profile_export_aiperf.json", recursive=True)):
r = json.load(open(p))
md = r.get("metadata", {})
print(f"### AGENTX_RESULT {p}")
print("submission_valid:", md.get("submission_valid"), md.get("submission_invalid_reasons"))
for k in ("request_throughput", "output_token_throughput", "time_to_first_token",
"inter_token_latency", "request_latency", "request_count", "error_request_count",
"context_overflow_count", "input_sequence_length", "output_sequence_length"):
v = r.get(k)
if isinstance(v, dict):
print(f"{k:28s} avg={v.get('avg')} p50={v.get('p50')} p90={v.get('p90')} p99={v.get('p99')} unit={v.get('unit')}")
elif v is not None:
print(f"{k:28s} {v}")
EOF

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If no benchmark profile export files are found (e.g., if the benchmark failed to run or output results), the python script currently exits silently with status 0. Adding a check to ensure results were found and using standard Python file context managers (with open) makes the reporting step much more robust.

          python3 - <<'EOF'
          import glob, json, sys
          results_found = False
          for p in sorted(glob.glob("/results/artifacts-c*/**/profile_export_aiperf.json", recursive=True)):
              results_found = True
              with open(p, 'r', encoding='utf-8') as f:
                  r = json.load(f)
              md = r.get("metadata", {})
              print(f"### AGENTX_RESULT {p}")
              print("submission_valid:", md.get("submission_valid"), md.get("submission_invalid_reasons"))
              for k in ("request_throughput", "output_token_throughput", "time_to_first_token",
                        "inter_token_latency", "request_latency", "request_count", "error_request_count",
                        "context_overflow_count", "input_sequence_length", "output_sequence_length"):
                  v = r.get(k)
                  if isinstance(v, dict):
                      print(f"{k:28s} avg={v.get('avg')} p50={v.get('p50')} p90={v.get('p90')} p99={v.get('p99')} unit={v.get('unit')}")
                  elif v is not None:
                      print(f"{k:28s} {v}")
          if not results_found:
              print("Error: No benchmark profile export files found!", file=sys.stderr)
              sys.exit(1)
          EOF

@Chris113113
Chris113113 merged commit 5070b47 into AI-Hypercomputer:main Oct 19, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants