docs(examples): GLM-5.2-FP8 PD on gfx942 — the docker recipe and its Kubernetes translation by llying-001 · Pull Request #96 · AMD-AGI/Infera

llying-001 · 2026-08-06T11:29:07Z

Description

GLM-5.2-FP8 was brought up, tuned and benchmarked on 2 × MI300X with SGLang
prefill/decode disaggregation, and nothing in the repo carried that deployment.
Anyone repeating it had to re-derive it, and the parts easiest to get wrong are
exactly the parts that fail quietly — a KV transfer that falls back to TCP, an L3
tier that is written and never read, kv-aware routing degraded to load balancing.

This lands the deployment twice: as the docker + shell scripts it was actually
validated with, and as the Kubernetes manifest that translates them. Every engine
and kvd flag in the manifest is the docker recipe's; section 6 of the recipe README
enumerates each place the substrate forced a difference, so a reader can check the
claim rather than trust it.

The values are the substance of this PR. They are not SGLang defaults, and each one
carries the measurement that chose it, taken one axis at a time on an agentic
multi-turn trace (32 conversations / 225 turns, CONC=16, ~68k-token median input):

Setting	Value	What it bought
`ROUTER_BACKEND`	`rust`	same routing decisions as `python`, 27% faster end to end
`CHUNK`	`8192` (1,024/rank)	−23.8% duration, −34.4% TTFT against 16,384/rank
MTP steps/topk/draft	`5`/`1`/`6`	−8.5% duration, −13.8% TPOT; 4.64 acceptance on a 4.00 break-even
`IB_DEVICE`	one rail	striping over every NIC measured 11.9% slower
dp-attention	both legs	pure TP8 prefill measured 25.9% slower
`KVD`	`0`	the offload tier cost 12% and served zero reads on this trace

Together: 764 s → 420 s (−45%), output throughput 64.8 → 118.0 tok/s. One metric
moved the wrong way — ITL p90 rose 13.7%, because a deeper draft emits tokens in
burstier groups.

Type of change

Documentation change (change only to the documentation, either a fix or a new content)
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Infra/Build change
Code refactoring

Nothing outside examples/ changes.

Changes

examples/recipes/glm5.2-fp8-gfx942/ — new Kubernetes recipe. The deployment
as one InferaDeployment: router + prefill (with a kvd sidecar) + decode, TP8/DP8
with DP-attention, MTP, fp8 KV, Mooncake RDMA between the legs, KV offload to host
RAM and node-local NVMe. kvd runs as a native sidecar (initContainers +
restartPolicy: Always, hence k8s 1.29+): the engine probes the kvd socket once
with a 5 s timeout and refuses to start if nothing answers, so "kvd first" has to
be a scheduling guarantee rather than a convention.
The recipe's --io-mode direct rationale, corrected. The kvd storage
classifier now falls back to a major:minor walk through sysfs when lsblk cannot
name a device, which is the exact failure the recipe cited as its reason for
pinning the flag. The flag stays — it is the validated value and a
misclassification is a silent 4× — but it is a pin now, not a workaround.
examples/glm5.2_gfx942/ — new docker + shell recipe. env.sh plus five
launch scripts (etcd, optional kvd daemon, both SGLang legs, the Infera router),
then verify.sh and bench.sh. Every failure this stack produces silently gets an
assertion: an arithmetic answer only reachable through an intact prefix, a non-zero
block count in the router's own pick log, spec_accept_length on the decode leg.
kvd ships here but defaults to KVD=0, which is the measured-best shape for this
workload rather than a hedge; env.sh refuses a KVD value that is neither 0 nor
1, since KVD=true would otherwise skip the daemon and produce a baseline run
wearing the kvd name.
The manifest's chunk and MTP shape, corrected. It landed with the docker
recipe's pre-sweep values while claiming they were identical, and both are the
sweep's largest levers: --chunked-prefill-size 131072 → 8192 (16,384 → 1,024 per
rank, since DP-attention splits the value dp_size ways) and EAGLE 3/1/4 → 5/1/6,
on both legs, because SGLang rejects a disaggregated pair whose speculative configs
differ. Section 5's kvd A/B also stops saying the measurement is pending: KVD=1
ran 12.0% slower with gets_total = 0 — 100.8 GB written to L3 and not one page
read back — because the 54 GB/rank device pool had already taken ~100% of the reuse
that trace offered.

Out of scope, deliberately: the agentic multi-turn benchmark that produced the
figures above. bench.sh here is sglang.bench_serving on random prompts, which
sizes the deployment and reproduces none of them; the trace harness is being
generalised into a standalone tool for a later PR.

Validation

Run against the live 2 × MI300X deployment:

verify.sh — 5/5 checks pass (both legs registered, correct answer through the PD
pair, 1561 blocks hashed on the largest router pick, spec_accept_length present
on the decode leg, Mooncake RDMA transport in the decode log).
bench.sh — completes and writes results/<tag>.{json,log}; the ~0% cache-hit
line is expected on a random dataset and is documented as such.
kvd paths — a real daemon brought up by launch/launch_kvd.sh on a scratch socket
and directory with small quotas: readiness probe and io_mode readout both behave.
KVD=0 is a clean no-op, KVD=true is rejected with exit 1, and KVD=1 without a
daemon makes launch_prefill.sh exit before it touches the running engines.
All 12 scripts pass bash -n; trailing whitespace, end-of-file newlines and file
sizes are clean against the repo's pre-commit hooks.

Checklist:

The functionality is complete
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works — N/A, no library code changes; validation is above
New and existing unit tests pass locally with my changes — N/A for the same reason

A `docker` + shell deployment of GLM-5.2-FP8 on 2 x MI300X was brought up and benchmarked, but nothing carried it into Kubernetes. Anyone repeating it had to re-derive the manifest from four launch scripts, and the parts that are easiest to get wrong there are exactly the parts that fail quietly -- a KV transfer that falls back to TCP, an L3 tier that is written and never read, a hicache page silently rejected for being larger than its tablespace slot. This is that deployment as one InferaDeployment: router + prefill (with a kvd sidecar) + decode, TP8/DP8 with DP-attention, MTP, fp8 KV, Mooncake RDMA between the legs, and KV offload to host RAM and node-local NVMe. Every engine, router and kvd flag is the docker recipe's, unchanged. The README's section 6 lists each place the substrate forced something different and why, so a reader can check the claim rather than trust it. Three of those are worth calling out because they are not mechanical: kvd runs as a NATIVE sidecar (initContainers + restartPolicy: Always, hence k8s 1.29+). The engine probes the kvd socket once with a 5 s timeout and refuses to start if nothing answers -- it does not retry -- so "kvd first" has to be a scheduling guarantee, not a convention. An ordinary container makes it a race. The kvd sidecar gets 136Gi, not the 64G its --max-bytes suggests. kvd holds two independent budgets: --max-bytes caps the inline store, while the shared arena is sized separately, defaults to the same value, and is mmap'd and mlocked whole at startup. Sizing the limit to one of them gets the sidecar OOM-killed mid-run, which reads as a kvd bug. The RDMA rail and GID index are placeholders, not the validated cluster's mlx5_0 and 3. leg.sh takes both as required variables with no fallback, and the GID index is documented as per-node -- so there are two of those, one per leg. Hard-coding them would have been less faithful to the recipe, not more. This recipe also does not use the overlay, which every other one does. GLM-5.2 on the v0.5.16 gfx942 base needs a rebuilt Mooncake engine.so and four SGLang source patches; the payload carries patches/vllm/ only and its patch loop is gated on `import vllm`, so on an SGLang base it does not run at all. Section 1 covers what breaks without each patch and how to read the markers out of a built image. Not yet run in its Kubernetes form. The validation table says so per row rather than in a footnote, and the two docker-side numbers still being re-measured are marked TODO instead of quoted. Signed-off-by: liyingli <liyingli@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>

… bug Rebasing onto main brought in f9a6a12, which teaches kvd's storage classifier to fall back to a major:minor walk through sysfs when lsblk cannot name a device. That is the exact failure the recipe cites as its reason for pinning --io-mode direct: an LVM mount seen from inside a container had no name lsblk could open, so the probe answered "unknown device" and the region took the conservative buffered branch. sysfs is indexed by numbers rather than names and the block layer is not namespaced, so auto reaches direct on its own now -- including from an unprivileged container, which is what the kvd sidecar is. The flag stays. It is the value the docker recipe ran and therefore the validated one, and a misclassification is a silent 4x on write-back, so pinning it is worth the line. Only the justification changes: it was a workaround, it is now a pin, and a reader who checks the classifier against the old wording would find they disagree. Left alone: the KVD_L3_DIR row's claim that a shared filesystem classifies as buffered. Nothing in that path resolves to a block device at all, so the sysfs fallback does not reach it. Also left alone is the "LVM-over-7-NVMe" figure -- it traces to the bench that produced the 3.70 / 14.56 GB/s pair on this same gfx942 mount, and the 8-NVMe rig in f9a6a12's message is where that fix was verified, not this node. Signed-off-by: liyingli <liyingli@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>

The Kubernetes recipe under examples/recipes/glm5.2-fp8-gfx942/ lifts every flag from a `docker` + shell deployment of GLM-5.2-FP8 on 2 x MI300X, and then had nothing to point at: the scripts that deployment was tuned with lived in a working directory. A reader could see the values but not how they were reached, and could not run the thing the manifest is a translation of. This is that deployment: env.sh plus five launch scripts for etcd, an optional kvd daemon, the two SGLang legs and the Infera router, then verify.sh and bench.sh. Every value in env.sh is the measured one rather than SGLang's default, and carries the measurement that chose it -- chunked prefill at 1,024/rank (-23.8% duration, -34.4% TTFT against 16,384/rank), MTP 5/1/6 (-8.5% duration at 4.64 acceptance on a 4.00 break-even), the rust router (same routing decisions, 27% faster end to end), one RDMA rail (striping measured 11.9% slower), dp-attention on both legs (pure TP8 prefill 25.9% slower). Together, -45% on the trace they were tuned against. Two things shaped the scripts more than the flags did. Every interesting failure in this stack is silent. A KV hand-off that drops the prefix returns fluent text about something else, not an HTTP error; kv-aware without a tokenizer routes on load and looks healthy; a disagg pair that disagrees on the MTP shape just stops speculating. So verify.sh asserts an arithmetic answer only reachable through an intact prefix, greps the router's own pick log for a non-zero block count, and reads spec_accept_length off the decode leg -- five checks, each aimed at one of those, exiting non-zero rather than printing. kvd ships here but defaults to KVD=0, which is the measured-best shape for this workload and not a hedge: the tier cost 12% and served zero reads, the 54 GB/rank device pool already answering ~100% of the reuse the trace had. README section 6 says when to turn it on and how to tell, and env.sh refuses a KVD value that is neither 0 nor 1 -- KVD=true would otherwise skip the daemon and produce a baseline run wearing the kvd name. bench.sh is deliberately the simple one: sglang.bench_serving on random prompts, which sizes the deployment and reproduces none of the figures above. The agentic multi-turn trace that produced them is being generalised into a standalone tool and is not part of this. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: liyingli <liyingli@amd.com>

…manifest The manifest landed with the docker recipe's values as they stood before its tuning sweep finished, and claimed they were identical. Two are not, and both are the sweep's largest levers. --chunked-prefill-size 131072 is 16,384 per rank under DP-attention, which splits the value dp_size ways. 8192 -- 1,024 per rank -- measured 23.8% lower duration and 34.4% lower TTFT on the trace both recipes are tuned against. The aggregate-vs-per-rank division is also why the number looks alarming either way, so the README now shows how to read back what actually took effect from /get_server_info rather than trusting the engine's "adjusted from ... to ..." line. EAGLE 3/1/4 -> 5/1/6, worth 8.5% duration and 13.8% TPOT. Each extra draft step costs 6.53 ms here, so the shape pays only while acceptance outruns it: 5 steps accept 4.64 against a 4.00 break-even, and 7/1/8 both misses its break-even and runs prefill out of activation memory. Applied to both legs -- SGLang rejects a disaggregated pair whose speculative configs differ. Section 5's kvd A/B no longer says the measurement is pending: KVD=1 ran 12.0% slower with gets_total = 0, 100.8 GB written to L3 and not one page read back, because the 54 GB/rank device pool had already taken ~100% of the reuse that trace offered. That is a statement about the workload, and the section now says so along with what to check before paying for the tier. Section 6 also stops claiming every router flag is identical; the manifest runs --router-backend python because the Rust binary supports only etcd discovery while the operator's is kubernetes, which is a substrate translation like the rest and now sits with them. The intro links the docker recipe it is a translation of, now that examples/glm5.2_gfx942/ gives it somewhere to point. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: liyingli <liyingli@amd.com>

Copilot

🟡 Changes recommended

There are a couple of concrete doc/config consistency issues (validation-status wording and a risky silent default in env.sh) that should be corrected to avoid misleading or misconfiguring users.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

Adds a fully-specified GLM-5.2-FP8 (gfx942 / MI300X-class) prefill/decode-disaggregated deployment recipe to examples/, including both the validated Docker bring-up scripts and a Kubernetes manifest translation intended to preserve the same engine/router/kvd settings.

Changes:

Added a new Kubernetes recipe (InferaDeployment) for GLM-5.2-FP8 gfx942 disaggregated PD with infera-kvd sidecar and RDMA KV transfer.
Added a new Docker + shell bring-up package (env + launch + verify + bench + stop) for the same topology and tuned settings.
Updated the recipes index/readme to reference the new recipe and clarify “combo shape” availability for pinned/validated configs.

File summaries

File	Description
examples/recipes/README.md	Adds the GLM-5.2-FP8 (gfx942) recipe entry and clarifies combo-shape expectations.
examples/recipes/glm5.2-fp8-gfx942/README.md	New Kubernetes recipe documentation, prerequisites, and docker→k8s translation notes.
examples/recipes/glm5.2-fp8-gfx942/disaggregated-kvd/deploy.yaml	New `InferaDeployment` manifest for router + prefill(+kvd sidecar) + decode on separate nodes.
examples/glm5.2_gfx942/README.md	New end-to-end Docker bring-up documentation for the gfx942 PD deployment.
examples/glm5.2_gfx942/env.sh	New shared configuration for all scripts (tuned engine/router/kvd settings).
examples/glm5.2_gfx942/host_container.sh	New host-side helper to create/manage the long-lived engine container with correct mounts/ulimits.
examples/glm5.2_gfx942/build_image.sh	New host-side helper to build the required patched engine image.
examples/glm5.2_gfx942/preflight_rdma.sh	New host-side RDMA visibility + optional cross-node fabric preflight.
examples/glm5.2_gfx942/launch/launch_etcd.sh	New host-side launcher for etcd used by the Docker PD deployment.
examples/glm5.2_gfx942/launch/launch_kvd.sh	New in-container launcher for optional `infera-kvd` daemon.
examples/glm5.2_gfx942/launch/launch_prefill.sh	New in-container launcher for the SGLang prefill leg (DP-attention, MTP, RDMA).
examples/glm5.2_gfx942/launch/launch_decode.sh	New in-container launcher for the SGLang decode leg matching prefill settings.
examples/glm5.2_gfx942/launch/launch_router.sh	New in-container launcher for the kv-aware router (rust/python backend selection).
examples/glm5.2_gfx942/verify.sh	New verification script to assert correctness + kv-aware steering + MTP + optional kvd writes.
examples/glm5.2_gfx942/bench.sh	New simple throughput sizing benchmark wrapper (random dataset) through the router.
examples/glm5.2_gfx942/stop.sh	New in-container shutdown script to stop engines/router and avoid VRAM relaunch races.
examples/glm5.2_gfx942/.gitignore	Ignores generated logs/ and results/ for the new example package.

Review details

Files reviewed: 17/17 changed files
Comments generated: 2
Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

+export PREFILL_IP="${PREFILL_IP:-$(getent ahostsv4 "$PREFILL_NODE" 2>/dev/null | awk 'NR==1{print $1}')}"
+export DECODE_IP="${DECODE_IP:-$(getent ahostsv4 "$DECODE_NODE" 2>/dev/null | awk 'NR==1{print $1}')}"
+: "${PREFILL_IP:=127.0.0.1}"
+: "${DECODE_IP:=$PREFILL_IP}"


+
+| What | Status |
+|---|---|
+| This configuration in its **`docker` form** | brought up and benchmarked on 2 × MI300X. **TODO — numbers withheld pending a re-run.** Request count, cache efficiency and the kvd counters are being re-measured on one consistent run; quote nothing here until they land |


…g to loopback env.sh fell back to 127.0.0.1 when a node name did not resolve, and DECODE_IP fell back to PREFILL_IP on top of that. Both are the wrong shape of default for a deployment whose two halves find each other only through the addresses they register. The full fallback hides itself on the node you are watching: etcd runs on the prefill node, so 127.0.0.1:2379 resolves there, the prefill leg registers, the router comes up, and nothing on that node looks wrong. The decode leg is the one that fails, on the other machine, and not until 20 minutes in -- registration happens after the weights load and the graphs capture. Setting only PREFILL_IP is the likelier trap and the worse one. DECODE_IP silently becomes the prefill node's address, the decode leg advertises it, and both legs register: the worker count verify.sh checks first is correct, and only a real request finds that nothing is listening at the address decode published. So require_ips(), called by the six scripts that dial one of these addresses and by no other. A blanket check in env.sh would have been shorter and wrong: all twelve scripts source it, but build_image.sh and host_container.sh need no address at all, and the README has the image build (§1.3) before the cluster adaptation (§2) that sets them. Failing there would trade this footgun for one in the first documented step. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: liyingli <liyingli@amd.com>

…gates The row said the docker-form figures were being re-measured and that nothing should be quoted until they landed. They landed, and the section 5 rewrite earlier on this branch quotes them -- including the kvd counters the row names explicitly. A reader hitting both has no way to tell which one is current. State what is actually true instead: the figures come from the sweep, and the kvd counters from its single KVD=1 run. The other two rows are unchanged and still say "not run", because the Kubernetes form still has not been. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: liyingli <liyingli@amd.com>

kzjeef

please add one k8s config receipt too.

jiejingzhangamd

please add the k8s config too.

llying-001 and others added 4 commits August 6, 2026 11:20

Copilot AI lite review requested due to automatic review settings August 6, 2026 11:29

llying-001 requested review from JohnQinAMD, jiejingzhangamd, limou102 and xiaobochen-amd as code owners August 6, 2026 11:29

Copilot started reviewing on behalf of llying-001 August 6, 2026 11:30 View session

Copilot AI reviewed Aug 6, 2026

View reviewed changes

llying-001 and others added 2 commits August 7, 2026 03:28

kzjeef approved these changes Aug 7, 2026

View reviewed changes

jiejingzhangamd approved these changes Aug 7, 2026

View reviewed changes

jiejingzhangamd merged commit 37e343d into main Aug 7, 2026
24 checks passed

Navigation Menu

Uh oh!

docs(examples): GLM-5.2-FP8 PD on gfx942 — the docker recipe and its Kubernetes translation - #96

docs(examples): GLM-5.2-FP8 PD on gfx942 — the docker recipe and its Kubernetes translation#96
jiejingzhangamd merged 6 commits into
mainfrom
llying/dev/glm5p2_fp8_docs

llying-001 commented Aug 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

kzjeef left a comment

Uh oh!

jiejingzhangamd left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

llying-001 commented Aug 6, 2026

Description

Type of change

Changes

Validation

Checklist:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

kzjeef left a comment

Choose a reason for hiding this comment

Uh oh!

jiejingzhangamd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants