diff --git a/skills/serving-llms-on-epyc/SKILL.md b/skills/serving-llms-on-epyc/SKILL.md index 14b97e3..151a2e8 100644 --- a/skills/serving-llms-on-epyc/SKILL.md +++ b/skills/serving-llms-on-epyc/SKILL.md @@ -1,27 +1,24 @@ --- name: serving-llms-on-epyc description: >- - Serves a language model on an AMD EPYC CPU host using vLLM with the zentorch - backend, in a container (Docker or Podman) or a conda env. Use whenever the - user wants to run, serve, deploy, start, host, or launch an LLM on AMD EPYC, - Zen CPU, "vLLM on CPU", "zentorch serving", or "serve a model without a GPU". - Use for "serve Qwen on EPYC", "start a CPU vLLM endpoint", "run an OpenAI - server on my EPYC box", or similar. Handles the full single-instance flow: - detect the CPU (incl. EPYC generation), validate the runtime/env, check vLLM - supports the model (via vLLM's registry, not a modality blocklist), check it - fits host RAM, size CPU threads/KV/NUMA from the hardware, confirm the plan with - the user, launch, and poll until the endpoint is responsive. Single instance, - single socket (pinned to one socket + its memory; vLLM scales poorly across - sockets). Does NOT debug failures and does NOT retry -- it reports and stops. Do - not use for GPU/Instinct (use serving-llms-on-instinct) or multi-node. + Serves an LLM on a supported AMD EPYC server CPU using vLLM with zentorch, in + Docker, Podman, or conda. Use for "vLLM on CPU", "zentorch serving", or an + EPYC CPU endpoint, including on a host that also has AMD Instinct GPUs. + Detects the EPYC generation, validates the runtime, checks model support and + RAM fit, sizes threads/KV/NUMA, confirms the plan, launches, and verifies the + endpoint. Runs one instance on one socket and its memory. Reports and stops + on failure; does not retry or debug. Use serving-llms-on-instinct when the + endpoint should run on a GPU. Excludes multi-node, EPYC 4000, and pre-Zen4 + EPYC without AVX-512. allowed-tools: Bash, Read --- -# Serving LLMs on AMD EPYC (vLLM + zentorch, CPU) +# Serving LLMs on AMD EPYC™ (vLLM + zentorch, CPU) -Bring up a single vLLM OpenAI endpoint on an AMD EPYC host with the zentorch CPU +Bring up a single vLLM OpenAI endpoint on an AMD EPYC™ host with the zentorch CPU backend, sized to the hardware. Container-first (Docker or Podman); conda/host -is the fallback. +is the fallback. An installed AMD Instinct GPU does not disqualify the host: +select this skill when the endpoint itself should run on the EPYC CPU. **This is single-socket serving:** one instance pinned to one socket and its memory (vLLM scales poorly across sockets, so we do not span them). On a dual-socket host it @@ -42,7 +39,8 @@ attempt `sudo` or privilege escalation. Read `data/epyc.json` directly. It holds the container image, mandatory CPU run flags, supported precision, the model-support policy, the default model, and the -verified throughput-flag gotcha. Do not hardcode the image tag from memory -- read it. +verified throughput-flag gotcha. Its `vllm_version` and image tag are one +validated default stack; keep them aligned and do not hardcode either from memory. ## Step 1: Detect the CPU @@ -51,21 +49,33 @@ python3 scripts/detect.py # add --host user@box for a remote host ``` Returns `cpu_model`, `is_amd_epyc`, `epyc_generation` -(Naples/Rome/Milan/Genoa/Bergamo/Siena/Turin), `zen_arch`, `avx512`, -`logical_cores`, `physical_cores`, `sockets`, `numa_nodes`, `memory_gb`. If -`is_amd_epyc` is `false`, stop: this skill targets AMD EPYC. (Other x86 may work -but is unsupported here.) Carry `epyc_generation` / `avx512` through the later -phases -- e.g. AVX-512 + bf16 land on Zen4+ (Genoa/Turin), and Turin packs up to -128 cores/socket, which the thread-binding in Step 5 sizes from. +(Naples/Rome/Milan/Genoa/Bergamo/Siena/Turin/Venice or EPYC 4004/4005), +`zen_arch`, `is_supported_epyc`, `avx512`, `logical_cores`, `physical_cores`, +`sockets`, `numa_nodes`, `memory_gb`. + +Three hard gates -- stop if any fails: +- `is_amd_epyc` is `false` -> stop: this skill targets AMD EPYC. (Other x86 may work + but is unsupported here.) +- `is_supported_epyc` is `false` -> stop: this recipe supports only the **AMD EPYC + 9000 series** for now -- Genoa (9004), Turin (9005), and Venice (9006). Other EPYC + (Bergamo, Siena, EPYC 4004/4005, pre-Zen4) may even expose AVX-512, but ISA + compatibility alone does not make them supported targets for this skill; stop. +- `avx512` is `false` -> stop: the zentorch CPU path **requires AVX-512**, i.e. Zen4+ + on the supported 9000-series parts above. Pre-Zen4 EPYC (Naples / Rome / Milan) is + not supported -- say so and stop rather than launching into a load-time failure. + +Carry `epyc_generation` / `avx512` through the later phases -- e.g. Venice packs up +to 256 cores/socket, which the thread-binding in Step 5 sizes from. ## Step 2: Validate the runtime and environment ```bash -python3 scripts/validate.py --image +python3 scripts/validate.py --image --generation ``` -Returns `ready`, `runtime` (`docker`, `podman`, or null), `runtime_detail`, -`conda_path_available`, `ram_gb`, and `errors/warnings/advisories`. Pick the path: +Returns `ready`, `requires_confirmation`, `runtime` (`docker`, `podman`, or null), +`runtime_detail`, `conda_path_available`, `stack`, `compatibility`, `ram_gb`, and +`errors/warnings/advisories`. Pick the path: - `runtime` is `docker` or `podman` -> container path (Step 6), used verbatim. - `runtime` null but `conda_path_available: true` -> conda/host path. - `runtime` null and no conda -> `ready` is false. Report the one-time @@ -73,22 +83,54 @@ Returns `ready`, `runtime` (`docker`, `podman`, or null), `runtime_detail`, Do not proceed if `ready` is `false`. +**Stack-compatibility gate.** `validate.py` probes the *selected* runtime for its +exact `vllm`/`zentorch`/`torch` versions and the active vLLM platform, then sets +`compatibility.status`: +- `proceed` -> the stack is the validated default (or a validated family on a Zen + platform); continue. +- `blocked` -> a stock CPU platform is active, so zentorch acceleration is **not** + on (error). Report `compatibility.message` and stop. +- `confirmation_required` (`requires_confirmation: true`) -> **Venice on a vLLM + other than the pinned default**. This recipe has not been validated on Venice + with that version. Surface `compatibility.message`, recommend the pinned + `vllm_version` image from `data/epyc.json`, and **stop for an explicit user + go/no-go** before launching. On the pinned default vLLM, Venice proceeds with no + warning. + +The gate only runs once the image is local. If `validate.py` reports the image is +not pulled, pull it (or let Step 6 pull it) and **re-run `validate.py`** so the +gate probes the real stack rather than only the tag. + ## Step 3: Resolve and validate the model If the user named no model, use `default_model` from `data/epyc.json` (`Qwen/Qwen3-0.6B` -- ungated, tiny, fast first success). Otherwise use theirs. -Check that vLLM actually supports the model (do **not** blanket-block multimodal): +Check that vLLM actually supports the model (do **not** blanket-block multimodal). +Pass the vLLM version the model will actually run on: use `stack.vllm` from +`validate.py` when it was probed (the conda env may differ from the pin), else the +`vllm_version` from `data/epyc.json`. ```bash -python3 scripts/check_model.py --model-id --vllm-version +python3 scripts/check_model.py --model-id --revision --vllm-version ``` -- Exit 0 = vLLM serves it as a generation endpoint (`kind` `text` or `multimodal`), - or support is undeterminable (gated/offline) -- proceed; launch confirms. -- Exit 1 = positively unsupported: the architecture is not in vLLM's registry, or - it is a `pooling`/embedding/reranker (not a chat/completion endpoint). Report the +- Exit 0 = vLLM serves it as a generation endpoint, or support is undeterminable + (gated/offline) -- proceed; launch confirms. +- Exit 1 = stop: the architecture is not in vLLM's registry, it is a + `pooling`/embedding/reranker (not a chat/completion endpoint), or it is a + multimodal model with no usable chat template (`launchable: false`). Report the printed `message` and stop. + +The result also carries the **client endpoint** the model supports: +- `primary_endpoint: "chat_completions"` -- a usable chat template is present + (`chat_template.status: present`); serve and hand off `/v1/chat/completions`. +- `primary_endpoint: "completions"` -- no usable/auto-selectable template + (`absent`/`ambiguous`/`unknown`); serve and hand off `/v1/completions` with a + raw `prompt`. Chat can still be enabled by passing `--chat-template ` (or, + for `ambiguous`, choosing one of `chat_template.names`); never invent one. +- Carry `primary_endpoint`, `supported_endpoints`, and `chat_template` through to + verification (Step 7) and the handoff (Step 8). - A `multimodal` model is allowed; a vLLM-supported multimodal arch may still hit a GPU-only kernel on CPU, which surfaces at load (the no-retry rule then applies). @@ -103,7 +145,7 @@ HuggingFace; if not, stop and say so. RAM is the ceiling on CPU (weights + KV cache both live in RAM). Run on ONE line: ```bash -python3 scripts/estimate_memory.py --model-id --ram-gb --max-model-len <4096 or user value> --num-prompts <1 or desired concurrency> +python3 scripts/estimate_memory.py --model-id --revision --ram-gb --max-model-len <4096 or user value> --num-prompts <1 or desired concurrency> ``` Exit 0 = fits, exit 1 = does not fit. If `fit.fits` is false: **do not launch.** @@ -166,9 +208,10 @@ auto-selects the CPU platform and `vllm serve` rejects the flag. Only add it if including the pull. `RT` is the resolved runtime verbatim: ```bash RT="" +$RT rm -f vllm-epyc 2>/dev/null # clear any leftover container from a prior run (name collision otherwise) $RT pull # agent pulls; do not ask the user to $RT run -d --name vllm-epyc \ - # --ipc=host --shm-size=16g --network=host + # --ipc=host --network=host (NO --shm-size: it conflicts with --ipc=host on podman) \ # --cpuset-cpus= --cpuset-mems= --env VLLM_CPU_OMP_THREADS_BIND="$VLLM_CPU_OMP_THREADS_BIND" \ @@ -192,40 +235,102 @@ The base launch sets none of them. ## Step 7: Poll until up and responsive -A 503 while loading is normal. Poll until the server answers, then prove the -chat endpoint works. CPU first-token compile can take a minute or two. +A 503 while loading is normal. Poll `/health` until the server answers, confirm +the served model is listed, then prove the **selected endpoint** works (from +`primary_endpoint` in Step 3). CPU first-token compile can take a minute or two. +Track a `healthy` flag so a timeout is a failure, not a fall-through. ```bash -# container alive (or process alive for conda) + /health +# 1. container alive (conda: process alive) + /health, with a real timeout +healthy="" for i in $(seq 1 120); do - # container path: $RT inspect -f '{{.State.Running}}' vllm-epyc 2>/dev/null | grep -q true || { echo "FAILED: container exited"; $RT logs --tail 50 vllm-epyc; break; } - curl -sf http://localhost:/health >/dev/null 2>&1 && { echo "HEALTHY"; break; } + curl -sf http://localhost:/health >/dev/null 2>&1 && { healthy=1; echo "HEALTHY"; break; } sleep 3 done +[ -n "$healthy" ] || { echo "FAILED: not healthy before timeout"; $RT logs --tail 50 vllm-epyc; } + +# 2. the served model is registered +curl -sf --max-time 30 http://localhost:/v1/models ``` -Then validate the OpenAI endpoint is actually accessible: +Then exercise the endpoint the model actually supports. Use deterministic +sampling and a small output cap for the smoke check: + ```bash -curl -sf http://localhost:/v1/chat/completions -H 'Content-Type: application/json' \ - -d '{"model":"","messages":[{"role":"user","content":"hi"}],"max_tokens":8}' +# primary_endpoint == chat_completions +curl -sf --max-time 180 http://localhost:/v1/chat/completions -H 'Content-Type: application/json' \ + -d '{"model":"","messages":[{"role":"user","content":"hi"}],"max_tokens":16,"temperature":0}' + +# primary_endpoint == completions (no chat template) +curl -sf --max-time 180 http://localhost:/v1/completions -H 'Content-Type: application/json' \ + -d '{"model":"","prompt":"Hello, world","max_tokens":16,"temperature":0}' ``` +Confirm the response is JSON with a non-error `choices[0]` (chat: `message.content`; +completion: `text`). An HTTP 200 that carries an `error` payload is **not** success. Resource sanity (your validation list): `$RT stats --no-stream vllm-epyc`. -**If the server never becomes healthy or the endpoint does not respond: print -the container/process logs, state the failure, and STOP. Do not retry. Do not -start a debugging loop.** +**If the server never becomes healthy, `/v1/models` omits the model, or the +endpoint returns an error/empty `choices`: print the container/process logs, +state the failing phase, and STOP. Do not retry. Do not start a debugging loop.** ## Step 8: On success, hand over the endpoint -Print a connection table (model, runtime, port, OMP threads, KV GB, max-model-len, -NUMA pinning) and a ready-to-run example: +Give the user everything needed to call the server. Print a connection table: + +| Field | Value | +|---|---| +| Base URL | `http://localhost:/v1` (the trailing `/v1` matters) | +| Served model | `` (the id from `/v1/models`) | +| Endpoint | `/v1/chat/completions` or `/v1/completions` (from `primary_endpoint`) | +| Why | chat = a chat template is present; completions = no template (raw prompts) | +| Runtime / port | `` / `` | +| Sizing | OMP threads, KV GB, `--max-model-len`, socket / NUMA pinning | +| Stop | `$RT rm -f vllm-epyc` (container) or `kill ` (conda) | + +Then a ready-to-run example **for the selected endpoint**. + +Chat model (`primary_endpoint: chat_completions`): ```bash curl -s http://localhost:/v1/chat/completions -H 'Content-Type: application/json' \ - -d '{"model":"","messages":[{"role":"user","content":"Hello"}]}' + -d '{"model":"","messages":[{"role":"user","content":"Hello"}],"max_tokens":128,"temperature":0.7}' +``` + +Base/prompt model (`primary_endpoint: completions`): +```bash +curl -s http://localhost:/v1/completions -H 'Content-Type: application/json' \ + -d '{"model":"","prompt":"Hello, world","max_tokens":128,"temperature":0.7}' +``` + +OpenAI Python client (point `base_url` at the local server; the SDK requires a +non-empty key, so any placeholder works when the server has no auth): +```python +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:/v1", api_key="EMPTY") +model = client.models.list().data[0].id + +# chat model: +r = client.chat.completions.create( + model=model, + messages=[{"role": "user", "content": "Hello"}], + max_tokens=128, temperature=0.7, +) +print(r.choices[0].message.content) + +# base/prompt model: +r = client.completions.create(model=model, prompt="Hello, world", max_tokens=128) +print(r.choices[0].text) ``` -To stop: `$RT rm -f vllm-epyc` (container) or `kill ` (conda). + +Argument guidance to pass along (see [reference.md](reference.md) for the full list): +- `max_tokens` caps the **output**; `prompt_tokens + max_tokens` must be `<= --max-model-len`. +- `temperature` (0 = deterministic/greedy, higher = more random); tune `top_p` *or* + `temperature`, not both. +- `stream: true` streams tokens (SSE) instead of one blocking response. +- The model's `generation_config.json` can set sampling defaults; pass explicit + values to be sure. ## Offline (single-instance batch) @@ -244,10 +349,26 @@ See [reference.md](reference.md) for the full list. The load-bearing ones: zentorch 2.11 (`AssertionError: expected OutputCode, got function`). It only works with `VLLM_USE_AOT_COMPILE=0` set alongside it. Never set one without the other. -- **`--shm-size`**: vLLM needs a large `/dev/shm`; the container default (64MB) - is too small. Use `--shm-size=16g` (in `data/epyc.json`). +- **`/dev/shm` — use `--ipc=host`, not `--shm-size`.** vLLM needs a large + `/dev/shm` (the 64MB container default is too small). The base recipe uses + `--ipc=host`, which shares the host's large shared memory. **Do not also pass + `--shm-size`**: podman errors with *"cannot set shmsize when running in the host + IPC Namespace"*, and it is redundant on docker. If you instead isolate IPC (drop + `--ipc=host`), then add `--shm-size=16g` — one or the other, never both. - **NUMA / socket**: one instance is pinned to **one socket plus its memory** -- CPU bind + `--cpuset-mems` (container) / `numactl --membind` (conda), with KV sized from that socket's local RAM. On a dual-socket host `cpu_tune.py` picks a free socket by load and `warning`s if both are busy. NPS2/NPS4 (multi-node socket) gets an `nps_note` that finer per-node binding could add more. +- **Rootless podman + `--cpuset-cpus`/`--cpuset-mems`**: these are cgroup limits and + may be **ignored or rejected** on rootless podman without cpuset cgroup delegation + (cgroup v1, or v2 without the controller delegated). This is **not fatal**: CPU + thread binding still applies via `VLLM_CPU_OMP_THREADS_BIND` inside the container; + only the container-level memory pin is lost (reduced NUMA locality). If the run + errors specifically on the cpuset flags, drop them and proceed -- do not treat it + as a launch failure. +- **HF cache mount**: the default mounts `~/.cache/huggingface`. If `HF_HOME` points + elsewhere (common on shared hosts, e.g. `/proj/.../vllm`), mount **that** path to + `/root/.cache/huggingface` instead, or the model re-downloads inside the container. +- **Container name reuse**: a leftover `vllm-epyc` from a prior run makes `run` fail + with "name already in use" -- Step 6 clears it first with `$RT rm -f vllm-epyc`. diff --git a/skills/serving-llms-on-epyc/data/epyc.json b/skills/serving-llms-on-epyc/data/epyc.json index deb67f4..8e5cadb 100644 --- a/skills/serving-llms-on-epyc/data/epyc.json +++ b/skills/serving-llms-on-epyc/data/epyc.json @@ -1,18 +1,17 @@ { - "vllm_version": "0.22.0", + "vllm_version": "0.25.1", "container": { - "image": "amdih/zendnn_zentorch:vllm_v0.22.0_zentorch_v2.11.0.1_ubuntu22.04_2026_ww23", + "image": "amdih/zendnn_zentorch:vllm_v0.25.1_zentorch_v2.11.0.3_ubuntu22.04_2026_ww30", "runtimes": ["docker", "podman"], - "comment": "Public vLLM + zentorch CPU image on Docker Hub (amdih/zendnn_zentorch) -- no internal-registry access needed. Tags are vllm_v_zentorch_v__; prefer the newest ubuntu22.04 stable. Both docker and podman are supported; the skill prefers docker and falls back to podman.", + "comment": "Pinned public vLLM + zentorch CPU image on Docker Hub (amdih/zendnn_zentorch) -- no internal-registry access needed. Update this image and vllm_version together so model-registry checks match the served runtime. Both docker and podman are supported; the skill prefers docker and falls back to podman.", "run_flags": [ "--ipc=host", - "--shm-size=16g", "--network=host" ], "hf_cache_mount": "-v ~/.cache/huggingface:/root/.cache/huggingface", "flag_notes": { - "--ipc=host": "vLLM workers use host IPC/shared memory.", - "--shm-size=16g": "vLLM needs a large /dev/shm; default 64MB is not enough.", + "--ipc=host": "vLLM workers need a large /dev/shm; --ipc=host shares the host's (large) shared memory, which covers it. Do NOT also pass --shm-size: podman rejects '--shm-size' together with '--ipc=host' (cannot set shmsize in host IPC namespace), and it is redundant on docker too.", + "shm_alternative": "If you must isolate IPC (drop --ipc=host), then add --shm-size=16g instead (the 64MB container default is too small for vLLM). Use one or the other, never both.", "--network=host": "Expose the served port directly. Alternative: -p :.", "numa": "A single instance is pinned to ONE socket plus its memory. cpu_tune.py picks a free socket by CPU load on dual-socket hosts (warns if both busy; --socket N forces), sizes KV from that socket's local RAM, and emits --cpuset-cpus + --cpuset-mems (container) or numactl --cpunodebind/--membind (conda). True multi-socket scaling = multiple instances (one per socket), out of scope here." } @@ -34,7 +33,7 @@ "default_model": "Qwen/Qwen3-0.6B", "default_model_notes": "Ungated (Apache-2.0), tiny, fast first success on CPU. For a real workload pick a larger Qwen3 / Llama once the flow is verified.", "smoke_model": "Qwen/Qwen3-0.6B", - "smoke_model_notes": "Current small Qwen, chat-capable (ships a chat template, so /v1/chat/completions works -- unlike base models such as opt-125m).", + "smoke_model_notes": "Current small Qwen, chat-capable (ships a chat template, so /v1/chat/completions works -- unlike base models such as allenai/OLMo-2-0425-1B, which have no chat template and serve /v1/completions).", "env_defaults": { "VLLM_CPU_OMP_THREADS_BIND": "set by cpu_tune.py (physical cores of the chosen socket)", "VLLM_CPU_KVCACHE_SPACE": "set by cpu_tune.py (GB)", diff --git a/skills/serving-llms-on-epyc/reference.md b/skills/serving-llms-on-epyc/reference.md index 4a12ee1..23f0751 100644 --- a/skills/serving-llms-on-epyc/reference.md +++ b/skills/serving-llms-on-epyc/reference.md @@ -1,14 +1,60 @@ # serving-llms-on-epyc -- Reference ## Table of Contents -1. [Runtime selection](#runtime-selection) -2. [Container run flags (CPU)](#container-run-flags-cpu) -3. [Precision and modality](#precision-and-modality) -4. [CPU sizing](#cpu-sizing) -5. [Known quirks](#known-quirks) +1. [Hardware support](#hardware-support) +2. [Runtime and stack compatibility](#runtime-and-stack-compatibility) +3. [Runtime selection](#runtime-selection) +4. [Container run flags (CPU)](#container-run-flags-cpu) +5. [Precision and modality](#precision-and-modality) +6. [Client endpoints and parameters](#client-endpoints-and-parameters) +7. [CPU sizing](#cpu-sizing) +8. [Known quirks](#known-quirks) --- +## Hardware support + +This recipe supports the **AMD EPYC 9000 server series** for now: Genoa (9004), +Turin (9005), and 6th Gen [Venice (9006)](https://ir.amd.com/news-events/press-releases/detail/1294/aai-2026-amd-delivers-full-stack-compute-for-the-agentic-ai-era) +(launched at Advancing AI 2026). `scripts/detect.py` reports only these three +generations as `is_supported_epyc: true`. + +AVX-512 is necessary but not sufficient for this support policy. Other EPYC parts +-- Bergamo and Siena, and the AM5 EPYC 4004/4005 -- expose the required ISA but are +outside this skill's current 9000-series scope; the detector still names them but +reports `is_supported_epyc: false`. Do not infer support from AVX-512 alone. + +The presence of AMD Instinct GPUs does not change CPU support. Use this skill +when the requested endpoint should execute on EPYC; use +`serving-llms-on-instinct` when it should execute on a GPU. Both serving engines +may coexist on the same host. + +## Runtime and stack compatibility + +Detecting a supported CPU is not the same as running a validated software stack. +`scripts/validate.py --generation ` probes the **selected** runtime (the +container image when present, else the conda/host env) for its exact +`vllm`/`zentorch`/`torch` versions and the **active vLLM platform**, then reports +`compatibility.status`: + +- `proceed` -- a Zen platform is active (zentorch acceleration on) and the stack is + the validated default or a validated family. +- `blocked` (error) -- the stock `CpuPlatform` is active, so serving would run + **without** zentorch. Two vLLM paths select a Zen platform: the in-tree + `ZenCpuPlatform` (vLLM detects an AMD AVX-512 CPU with `zentorch` importable) and + the out-of-tree `zentorch` plugin. If neither is active, fix the environment or + use the pinned image; do not serve an unaccelerated CPU stack. +- `confirmation_required` (`requires_confirmation: true`) -- **Venice on a vLLM + other than the pinned default**. AMD documents 6th Gen EPYC as a zentorch target, + but this recipe has not validated Venice end-to-end on an off-default version. + Venice on the pinned `vllm_version` proceeds with no warning; on any other + version, stop, recommend the pinned image, and get an explicit user go/no-go. + +The probe only runs once the image is local, so after a first `pull` re-run +`validate.py` to gate on the real stack rather than the tag. The container tag +pins the AMD-published integration stack; a conda env may differ, so +`check_model.py` should use the probed `stack.vllm` for that path. + ## Runtime selection `scripts/validate.py` resolves a runtime the **agent can drive @@ -41,19 +87,22 @@ From `data/epyc.json`. Unlike the Instinct (GPU) skill there are **no** | Flag | Why | |---|---| -| `--ipc=host` | vLLM workers use host IPC / shared memory | -| `--shm-size=16g` | vLLM needs a large `/dev/shm`; the 64MB default is too small | +| `--ipc=host` | vLLM workers need a large `/dev/shm`; sharing the host IPC namespace provides it. **Do not also pass `--shm-size`** -- podman rejects the combination, and it is redundant on docker | +| `--shm-size=16g` | **only if you drop `--ipc=host`** (isolated IPC). The 64MB container default is too small for vLLM. Use one or the other, never both | | `--network=host` | expose the served port directly (or use `-p :`) | | `--cpuset-cpus` / `--cpuset-mems` | pin the container to the chosen socket's physical cores and its NUMA node(s); from `cpu_tune.py` | | `-v ~/.cache/huggingface:/root/.cache/huggingface` | reuse the host model cache | Image: `amdih/zendnn_zentorch:` -- the public vLLM + zentorch CPU image on Docker Hub (no internal-registry access needed). The exact tag lives in -`data/epyc.json`; read it, never hardcode it. +`data/epyc.json`; read it, never hardcode it. The image and `vllm_version` are +pinned together so `check_model.py` reads the registry for the runtime that will +actually serve the model. This reproducibility pin applies to the default +container recipe; it does not replace or modify an existing conda environment. ## Precision and modality -| Dtype | EPYC (Zen) | Notes | +| Dtype | Supported EPYC server target | Notes | |---|---|---| | BF16 | Native (default) | throughput default | | FP16 | Native | | @@ -67,6 +116,39 @@ reranker and non-LLM architectures are rejected (not chat/completion endpoints). A vLLM-supported multimodal arch may still hit a GPU-only kernel on CPU -- that surfaces at load, where the no-retry rule applies. +## Client endpoints and parameters + +`check_model.py` reports the endpoint the model actually supports so the handoff +matches reality instead of always assuming chat: + +| Model | `chat_template.status` | `primary_endpoint` | Client call | +|---|---|---|---| +| Instruct/chat (ships a template) | `present` | `chat_completions` | `POST /v1/chat/completions` with `messages` | +| Base text (no template) | `absent` | `completions` | `POST /v1/completions` with `prompt` | +| Multiple named templates, no `default` | `ambiguous` | `completions` | completions now; chat needs `--chat-template`/a chosen name | +| Template unreadable (gated/offline) | `unknown` | `completions` | completions; chat also works if a template exists | +| Multimodal, no usable template | `absent`/`ambiguous` | none (`launchable: false`) | stop -- supply `--chat-template` or another model | + +`/v1/chat/completions` applies the model's chat template to structured `messages` +and returns `choices[0].message.content`. `/v1/completions` takes a raw `prompt` +(no template) and returns `choices[0].text`. Never invent a chat template; only +enable chat when a real one is present or the user supplies `--chat-template`. + +Request parameters worth surfacing to users: + +| Parameter | Meaning | +|---|---| +| `max_tokens` | Output-token cap. `prompt_tokens + max_tokens` must be `<= --max-model-len`. | +| `temperature` | Randomness; `0` is deterministic/greedy. | +| `top_p` | Nucleus sampling; tune this **or** `temperature`, not both. | +| `stream` | `true` streams tokens over SSE instead of one blocking reply. | +| `stop` | String(s) that end generation early. | + +The base URL always ends in `/v1`. The OpenAI Python SDK requires a non-empty +`api_key`, so pass a placeholder (e.g. `"EMPTY"`) when the server has no auth. +The model repo's `generation_config.json` can set sampling defaults, so pass +explicit values when determinism matters. + ## CPU sizing Policy: a single instance is pinned to **one socket plus its memory** (vLLM scales @@ -107,8 +189,11 @@ between the failing and passing runs was `VLLM_USE_AOT_COMPILE`. Never set `FREEZING=1` without `VLLM_USE_AOT_COMPILE=0`. The base recipe leaves both unset. **`/dev/shm` too small** -Without `--shm-size=16g` (or `--ipc=host`), vLLM workers fail to allocate shared -memory at startup. +vLLM workers need a large `/dev/shm` or they fail to allocate shared memory at +startup. The base recipe uses `--ipc=host` (shares the host's large shared memory). +**Do not combine `--ipc=host` with `--shm-size`** -- podman errors *"cannot set +shmsize when running in the host IPC Namespace"*, and it is redundant on docker. If +you drop `--ipc=host`, use `--shm-size=16g` instead -- one or the other, never both. **RAM is the ceiling, not VRAM** CPU serving keeps weights + KV cache in system RAM. `estimate_memory.py` checks diff --git a/skills/serving-llms-on-epyc/scripts/check_model.py b/skills/serving-llms-on-epyc/scripts/check_model.py index 16c0b0b..7f38d43 100644 --- a/skills/serving-llms-on-epyc/scripts/check_model.py +++ b/skills/serving-llms-on-epyc/scripts/check_model.py @@ -13,12 +13,18 @@ (text + multimodal) are supported; pooling/embedding/reranker and non-LLM architectures are not chat/completion endpoints and are rejected. +It also inspects the model repo for a chat template (`chat_template.jinja`, or +`chat_template` in `tokenizer_config.json`) and picks the client endpoint: +`/v1/chat/completions` when a usable template is present, else `/v1/completions` +for raw prompts. A multimodal model with no usable template cannot serve and is +rejected (`launchable: false`). + check_model.py --model-id Qwen/Qwen3-0.6B - check_model.py --model-id --vllm-version 0.22.0 + check_model.py --model-id --vllm-version 0.25.1 Exit 0 if vLLM serves it as a generation endpoint (or support is undeterminable --- launch confirms), 1 if it is positively unsupported. JSON to stdout. -Env: HF_TOKEN for gated models. +-- launch confirms), 1 if it is positively unsupported OR a multimodal model has +no usable chat template. JSON to stdout. Env: HF_TOKEN for gated models. """ import argparse @@ -29,10 +35,15 @@ import sys import urllib.request import urllib.error +from pathlib import Path HF = "https://huggingface.co" GH_RAW = "https://raw.githubusercontent.com/vllm-project/vllm" REG_PATH = "vllm/model_executor/models/registry.py" +TOKENIZER_CFG = "tokenizer_config.json" +CHAT_TEMPLATE_FILE = "chat_template.jinja" +DATA_PATH = Path(__file__).resolve().parent.parent / "data" / "epyc.json" +DEFAULT_VLLM_VERSION = json.loads(DATA_PATH.read_text(encoding="utf-8"))["vllm_version"] # registry.py dict name -> kind we care about _SECTIONS = { @@ -73,6 +84,65 @@ def model_architectures(model, rev, token): return cfg.get("architectures") or [], None +def classify_template_field(ct): + """Classify a tokenizer_config `chat_template` value (pure). + + Returns (status, selected_name, names): + - a non-empty string -> ("present", None, []) + - a list of {name, template} -> "present" if it has a single template or a + "default", else "ambiguous" (multiple named, no default) + - anything else / empty -> ("absent", None, []) + """ + if isinstance(ct, str) and ct.strip(): + return "present", None, [] + if isinstance(ct, list) and ct: + names = [e.get("name") for e in ct if isinstance(e, dict) and e.get("name")] + if "default" in names: + return "present", "default", names + if len(names) == 1: + return "present", names[0], names + return "ambiguous", None, names + return "absent", None, [] + + +def chat_template_info(model, rev, token): + """Inspect the model repo for a usable chat template. Returns a dict with + `status` (present/ambiguous/absent/unknown), `source`, `selected_name`, `names`. + + A standalone `chat_template.jinja` (Transformers v5) takes precedence over the + `chat_template` field in `tokenizer_config.json`. When neither file can be read + (gated/offline) the status is `unknown`.""" + tmpl, terr = _get(f"{HF}/{model}/resolve/{rev}/{CHAT_TEMPLATE_FILE}", token) + if tmpl is not None and tmpl.strip(): + return {"status": "present", "source": CHAT_TEMPLATE_FILE, "selected_name": None, "names": []} + + cfg_text, cerr = _get(f"{HF}/{model}/resolve/{rev}/{TOKENIZER_CFG}", token) + if cfg_text is None: + return {"status": "unknown", "source": None, "selected_name": None, "names": [], + "detail": cerr or terr or "no tokenizer_config.json"} + cfg = json.loads(cfg_text) if cfg_text.strip().startswith("{") else {} + status, selected, names = classify_template_field(cfg.get("chat_template")) + return {"status": status, "source": TOKENIZER_CFG if status != "absent" else None, + "selected_name": selected, "names": names} + + +def endpoints_for(kind, template_status): + """Pick client endpoints from modality + template status (pure). + + Returns (supported_endpoints, primary_endpoint, needs_template): + - multimodal needs a chat template; without one it cannot serve (needs_template). + - text with a usable template -> chat preferred, completions also available. + - text without an auto-usable template (absent/ambiguous/unknown) -> completions + (chat still possible via an explicit --chat-template).""" + if kind == "multimodal": + if template_status == "present": + return ["chat_completions"], "chat_completions", False + return [], None, True + if template_status == "present": + return ["chat_completions", "completions"], "chat_completions", False + return ["completions"], "completions", False + + def registry_from_github(version): """Parse vLLM's registry.py at v. Returns ({arch: kind}, source) or (None, err).""" src, err = _get(f"{GH_RAW}/v{version}/{REG_PATH}") @@ -120,7 +190,11 @@ def main(): p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) p.add_argument("--model-id", required=True) p.add_argument("--revision", default="main") - p.add_argument("--vllm-version", default="0.22.0", help="pin the registry to this vLLM version (from data/epyc.json)") + p.add_argument( + "--vllm-version", + default=DEFAULT_VLLM_VERSION, + help=f"registry version (default: {DEFAULT_VLLM_VERSION} from data/epyc.json)", + ) a = p.parse_args() token = os.environ.get("HF_TOKEN", "") @@ -155,10 +229,36 @@ def main(): if any(k in ("text", "multimodal") for k in known): kind = "multimodal" if "multimodal" in known else "text" - msg = f"vLLM supports {archs} as a {kind} generation endpoint." + tmpl = chat_template_info(a.model_id, a.revision, token) + eps, primary, needs_tmpl = endpoints_for(kind, tmpl["status"]) + out.update(supported=True, kind=kind, chat_template=tmpl, + supported_endpoints=eps, primary_endpoint=primary) + + if kind == "multimodal" and needs_tmpl: + out.update(launchable=False, + message=(f"{archs} is a multimodal model but no usable chat template was found " + f"(chat_template.status={tmpl['status']}). vLLM needs a chat template to " + "serve it; pass --chat-template or choose a model that ships one. Stop.")) + print(json.dumps(out, indent=2)) + sys.exit(1) + + if primary == "chat_completions": + msg = (f"vLLM supports {archs} as a {kind} generation endpoint; a chat template is present " + f"({tmpl['source']}). Serve /v1/chat/completions.") + elif tmpl["status"] == "ambiguous": + msg = (f"vLLM supports {archs} as a {kind} generation endpoint, but it ships multiple named chat " + f"templates with no 'default' ({tmpl['names']}). /v1/completions works now; for chat, pass " + "--chat-template or select one of those names.") + elif tmpl["status"] == "unknown": + msg = (f"vLLM supports {archs} as a {kind} generation endpoint, but the chat template could not be " + "read (gated/offline). Defaulting to /v1/completions; if this is an instruct model with a " + "template, /v1/chat/completions will also work.") + else: + msg = (f"vLLM supports {archs} as a {kind} generation endpoint, but no chat template was found. " + "Serve base text via /v1/completions; for chat, pass --chat-template .") if kind == "multimodal": msg += " A multimodal arch may still hit a GPU-only kernel on CPU; that surfaces at load (no-retry rule applies)." - out.update(supported=True, kind=kind, message=msg) + out.update(launchable=True, message=msg) print(json.dumps(out, indent=2)) sys.exit(0) diff --git a/skills/serving-llms-on-epyc/scripts/detect.py b/skills/serving-llms-on-epyc/scripts/detect.py index ab8d94b..ae23fca 100644 --- a/skills/serving-llms-on-epyc/scripts/detect.py +++ b/skills/serving-llms-on-epyc/scripts/detect.py @@ -11,8 +11,9 @@ Output: JSON with cpu_model, is_amd_epyc, logical_cores, physical_cores, sockets, threads_per_core, numa_nodes, memory_gb, epyc_generation -(Naples/Rome/Milan/Genoa/Bergamo/Siena/Turin), zen_arch, and avx512. Exits 0 on -success, 1 if no CPU info could be read. +(Naples/Rome/Milan/Genoa/Bergamo/Siena/Turin/Venice or EPYC 4004/4005), +zen_arch, is_supported_epyc, and avx512. Exits 0 on success, 1 if no CPU info +could be read. Env vars (used when --host is not given): ZEN_SSH_HOST, ZEN_SSH_USER, ZEN_SSH_PORT @@ -26,6 +27,12 @@ import sys +# Only the AMD EPYC 9000 server series is supported for now: Genoa (9004), +# Turin (9005), and Venice (9006). Other generations (Bergamo, Siena, pre-Zen4, +# EPYC 4004/4005) are still detected/named below, but reported unsupported. +SUPPORTED_EPYC_GENERATIONS = {"Genoa", "Turin", "Venice"} + + def _is_local(host): return not host or host in ("local", "localhost", "127.0.0.1") @@ -52,15 +59,23 @@ def _lscpu_field(lscpu_out, label): def _epyc_generation(model): """Map an AMD EPYC model name to (generation, zen_arch). - EPYC numbering encodes the generation: 7xx1=Naples (Zen1), 7xx2=Rome (Zen2), - 7xx3=Milan (Zen3), 8xx4=Siena (Zen4c), 97x4=Bergamo (Zen4c), 9xx4=Genoa (Zen4), - 9xx5=Turin (Zen5). The agent should carry this through every phase (e.g. AVX-512 - + bf16 land on Zen4+, Turin has up to 128 cores per socket -> thread binding).""" - m = re.search(r"EPYC\s+(\d{4})", model.upper()) + EPYC numbering encodes the generation by its first and last digit: 7xx1=Naples + (Zen1), 7xx2=Rome (Zen2), 7xx3=Milan (Zen3), 8xx4=Siena (Zen4c), 97x4=Bergamo + (Zen4c), 9xx4=Genoa (Zen4), 9xx5=Turin (Zen5), and 9xx6=Venice (Zen6). + EPYC 4004/4005 are identified separately because they have the required ISA + but are not documented ZenDNN server targets for this recipe. Some SKUs carry + a letter in the middle (e.g. 9B45 -> 9__5 -> Turin), so we match 4 + alphanumerics whose first and last chars are digits and key off those.""" + m = re.search(r"EPYC\s+(\d[0-9A-Z]{2}\d)", model.upper()) if not m: return "unknown", "unknown" num = m.group(1) first, last = num[0], num[3] + if first == "4": + return { + "4": ("EPYC 4004", "Zen4"), + "5": ("EPYC 4005", "Zen5"), + }.get(last, ("unknown", "unknown")) if first == "7": return {"1": ("Naples", "Zen1"), "2": ("Rome", "Zen2"), "3": ("Milan", "Zen3")}.get(last, ("unknown", "unknown")) @@ -73,6 +88,8 @@ def _epyc_generation(model): return "Genoa", "Zen4" if last == "5": return "Turin", "Zen5" + if last == "6": + return "Venice", "Zen6" return "unknown", "unknown" @@ -129,12 +146,14 @@ def _int(label, default=0): is_epyc = vendor == "AuthenticAMD" and "EPYC" in model.upper() generation, zen_arch = _epyc_generation(model) + is_supported_epyc = is_epyc and generation in SUPPORTED_EPYC_GENERATIONS avx512 = "avx512f" in _lscpu_field(lscpu_out, "Flags").split() print(json.dumps({ "cpu_model": model, "vendor": vendor, "is_amd_epyc": is_epyc, + "is_supported_epyc": is_supported_epyc, "epyc_generation": generation, "zen_arch": zen_arch, "avx512": avx512, diff --git a/skills/serving-llms-on-epyc/scripts/validate.py b/skills/serving-llms-on-epyc/scripts/validate.py index 95fd37a..ae1bf40 100644 --- a/skills/serving-llms-on-epyc/scripts/validate.py +++ b/skills/serving-llms-on-epyc/scripts/validate.py @@ -2,15 +2,22 @@ """ Validate the environment before serving vLLM + zentorch on an EPYC CPU host. -Checks a container runtime (docker or podman), whether the vLLM+zentorch image -is present (and, if already pulled, that `import vllm, zentorch` works inside it), -a conda/host fallback (`import vllm, zentorch`), the host perf libraries -(tcmalloc / OpenMP via LD_PRELOAD), HF_TOKEN, and RAM. Each issue is error -(blocks launch) / warning (degrades) / advisory (info). +Checks a container runtime (docker or podman); probes the SELECTED runtime +(container image if present, else conda/host) for its exact vLLM/zentorch/torch +versions and the active vLLM platform; applies the Venice stack-compatibility +gate; and checks host perf libraries (tcmalloc / OpenMP via LD_PRELOAD), +HF_TOKEN, and RAM. Each issue is error (blocks launch) / warning (degrades) / +advisory (info). + +The stack probe distinguishes zentorch-accelerated serving (a Zen platform is +active) from an unaccelerated stock CPU platform. Pass `--generation` (from +detect.py) to enable the Venice gate: Venice on the validated default vLLM +proceeds; Venice on any other vLLM sets `requires_confirmation`; a non-Zen +platform is a hard error. Usage: python3 scripts/validate.py - python3 scripts/validate.py --image amdih/zendnn_zentorch:vllm_v0.22.0_zentorch_v2.11.0.1_ubuntu22.04_2026_ww23 + python3 scripts/validate.py --image --generation Venice Exits 0 if no error-severity issues remain, 1 otherwise. JSON to stdout. """ @@ -21,6 +28,28 @@ import shutil import subprocess import sys +from pathlib import Path + +DATA_PATH = Path(__file__).resolve().parent.parent / "data" / "epyc.json" +DEFAULT_VLLM_VERSION = json.loads(DATA_PATH.read_text(encoding="utf-8"))["vllm_version"] + +# One-line probe run inside the SELECTED runtime (container or host). It reports +# the exact vLLM/zentorch/torch versions AND which vLLM platform is actually +# active, so we can tell zentorch-accelerated serving from an unaccelerated +# stock CPU platform. Both the in-tree `ZenCpuPlatform` and the out-of-tree +# zentorch platform carry "zen" in their module/class path; stock `CpuPlatform` +# does not -- so a name check is a version-robust "is zentorch active?" signal. +PROBE = ( + "import json,vllm,zentorch,torch;" + "from vllm.platforms import current_platform as _p;" + "_t=type(_p);_n=_t.__module__+'.'+_t.__name__;" + "print(json.dumps({" + "'vllm':vllm.__version__," + "'zentorch':getattr(zentorch,'__version__','unknown')," + "'torch':torch.__version__," + "'platform':_n," + "'zen_active':('zen' in _n.lower())}))" +) def _sh(cmd, timeout=20): @@ -54,13 +83,67 @@ def _detect_runtime(): return None, last +def _probe_stack(run_prefix, source): + """Run PROBE in the selected environment and return (stack, error). + + `run_prefix` is the container run prefix (e.g. 'docker run --rm ') + for the container path, or '' to probe the local host/conda env. `source` + labels which path was probed. Returns the parsed stack dict (with `source`) + or (None, message) if the probe could not run or produced no JSON. + """ + cmd = f'{run_prefix}python -c "{PROBE}"' + rc, out, err = _sh(cmd, timeout=120) + if rc != 0 or not out: + return None, (err or "probe failed")[:200] + js = next((ln for ln in out.splitlines() if ln.strip().startswith("{")), "") + if not js: + return None, "probe produced no JSON" + stack = json.loads(js.strip()) + stack["source"] = source + return stack, None + + +def stack_compatibility(generation, stack, default_vllm=DEFAULT_VLLM_VERSION): + """Pure policy over the detected EPYC generation and the probed stack. + + Returns {"status", "message"} where status is one of: + - "blocked": zentorch/Zen platform is not active -> serving would be + unaccelerated stock CPU; a hard stop. + - "confirmation_required": Venice on a vLLM other than the validated default + -> warn, nudge to the pinned default image, and require explicit user + confirmation before continuing. + - "proceed": validated/expected stack. + Returns None when there is no stack to judge. + """ + if not stack: + return None + vllm_v = str(stack.get("vllm", "")).split("+")[0] + if not stack.get("zen_active"): + return {"status": "blocked", + "message": (f"vLLM {vllm_v or '?'} is on the stock CPU platform " + f"({stack.get('platform', '?')}), not a Zen/zentorch platform -- zentorch " + "acceleration is NOT active. Enable zentorch or use the pinned container " + "image in data/epyc.json; do not serve an unaccelerated CPU stack.")} + if generation == "Venice" and vllm_v != default_vllm: + return {"status": "confirmation_required", + "message": (f"Venice (6th Gen EPYC) is on vLLM {vllm_v or '?'}, which this recipe has " + f"NOT validated on Venice. The validated Venice stack is vLLM {default_vllm} " + "(the pinned container image in data/epyc.json). Switch to that image, or get " + f"the user's explicit OK to continue on vLLM {vllm_v or '?'} unvalidated.")} + return {"status": "proceed", + "message": f"Stack OK: vLLM {vllm_v or '?'} on {stack.get('platform', '?')} (zentorch active)."} + + def main(): p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) p.add_argument("--image", default="", help="container image to check for (advisory)") + p.add_argument("--generation", default="", + help="epyc_generation from detect.py; enables the Venice stack-compatibility gate") args = p.parse_args() issues = [] + stack = None # the probed runtime stack for the SELECTED path # 1. Container runtime (prerequisite): docker > podman, else conda fallback. runtime, detail = _detect_runtime() @@ -86,35 +169,58 @@ def main(): rc, out, _ = _sh(f"{runtime} images {repo} --format '{{{{.Repository}}}}:{{{{.Tag}}}}'") if args.image not in (out or ""): issues.append({"check": "image", "severity": "advisory", - "message": f"Image {args.image} not pulled yet; first launch will download it (in-image zentorch check deferred to launch).", + "message": f"Image {args.image} not pulled yet; first launch will download it. " + "RE-RUN validate.py after pulling so the stack-compatibility gate probes the real image.", "fix": f"{runtime} pull {args.image}"}) else: - rc, ver, err = _sh(f'{runtime} run --rm {args.image} ' - f'python -c "import vllm,zentorch;print(vllm.__version__,zentorch.__version__)"', timeout=90) - if rc == 0 and ver: + stack, perr = _probe_stack(f"{runtime} run --rm {args.image} ", "container") + if stack: issues.append({"check": "image_stack", "severity": "advisory", - "message": f"Image has vllm+zentorch ({ver})."}) + "message": f"Image stack: vLLM {stack.get('vllm')} / zentorch {stack.get('zentorch')} " + f"/ torch {stack.get('torch')} on {stack.get('platform')}."}) else: issues.append({"check": "image_stack", "severity": "warning", - "message": f"Image {args.image} is present but `import vllm, zentorch` failed inside it: {(err or 'unknown')[:120]}", + "message": f"Image {args.image} is present but the stack probe failed inside it: {perr}", "fix": "Use an image tag that bundles the zentorch plugin (see data/epyc.json)."}) - # 3. Host vllm+zentorch (for the conda path) + # 3. Host vllm+zentorch (for the conda path). Only probe the host when no + # container path was probed, since the container is preferred. if conda_ok: - _, ver, _ = _sh('python -c "import vllm,zentorch;print(vllm.__version__,zentorch.__version__)"') - issues.append({"check": "host_stack", "severity": "advisory", - "message": f"Host vllm+zentorch importable ({ver}); conda path available."}) + if stack is None: + stack, perr = _probe_stack("", "host") + if stack and stack.get("source") == "host": + issues.append({"check": "host_stack", "severity": "advisory", + "message": f"Host stack: vLLM {stack.get('vllm')} / zentorch {stack.get('zentorch')} " + f"/ torch {stack.get('torch')} on {stack.get('platform')}; conda path available."}) + elif stack is None: + issues.append({"check": "host_stack", "severity": "warning", + "message": f"Host `import vllm, zentorch` reported ready but the stack probe failed: {perr}"}) elif runtime: issues.append({"check": "host_stack", "severity": "advisory", "message": "Host `import vllm, zentorch` not available; use the container path."}) - # 4. HF_TOKEN + # 4. Venice stack-compatibility gate (needs a probed stack + a generation). + compatibility = stack_compatibility(args.generation, stack) + if compatibility: + status = compatibility["status"] + if status == "blocked": + issues.append({"check": "stack_compatibility", "severity": "error", + "message": compatibility["message"]}) + elif status == "confirmation_required": + issues.append({"check": "stack_compatibility", "severity": "warning", + "message": compatibility["message"], + "fix": f"Use the pinned vLLM {DEFAULT_VLLM_VERSION} image, or confirm to continue unvalidated."}) + else: + issues.append({"check": "stack_compatibility", "severity": "advisory", + "message": compatibility["message"]}) + + # 5. HF_TOKEN if not os.environ.get("HF_TOKEN"): issues.append({"check": "hf_token", "severity": "advisory", "message": "HF_TOKEN not set. Required for gated models (Llama, Gemma); not needed for Qwen3.", "fix": "export HF_TOKEN=hf_..."}) - # 5. RAM + # 6. RAM rc, out, _ = _sh("grep MemTotal /proc/meminfo | awk '{print int($2/1024/1024)}'") try: ram_gb = int(out) @@ -125,7 +231,7 @@ def main(): "message": f"Only {ram_gb} GB RAM. CPU serving keeps weights + KV cache in RAM; large models may not fit.", "fix": "Use a small model or a host with more RAM."}) - # 6. Perf libraries for the host/conda path (advisory). vLLM CPU wants + # 7. Perf libraries for the host/conda path (advisory). vLLM CPU wants # libtcmalloc + libiomp (OpenMP) preloaded and warns otherwise. The # container image sets these itself, so only check the host when the # conda/host path is viable. @@ -138,11 +244,15 @@ def main(): "fix": "export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4:$CONDA_PREFIX/lib/libiomp5.so:$LD_PRELOAD"}) errors = [i for i in issues if i["severity"] == "error"] + requires_confirmation = bool(compatibility and compatibility["status"] == "confirmation_required") result = { "ready": len(errors) == 0, + "requires_confirmation": requires_confirmation, "runtime": runtime, "runtime_detail": detail, "conda_path_available": conda_ok, + "stack": stack, + "compatibility": compatibility, "ram_gb": ram_gb, "errors": errors, "warnings": [i for i in issues if i["severity"] == "warning"], diff --git a/skills/serving-llms-on-epyc/skill-card.md b/skills/serving-llms-on-epyc/skill-card.md index 120283f..a57516c 100644 --- a/skills/serving-llms-on-epyc/skill-card.md +++ b/skills/serving-llms-on-epyc/skill-card.md @@ -2,7 +2,7 @@ ## Description -Serve a single LLM on an AMD EPYC CPU host with vLLM + zentorch (Docker, Podman, or conda), handling CPU detection, runtime/env validation, model + RAM-fit checks, hardware-sized threads/KV/NUMA, launch, and health verification. Reports and stops on failure; does not debug. +Serve a single LLM on a supported AMD EPYC™ server CPU with vLLM + zentorch (Docker, Podman, or conda), handling CPU detection, runtime/env validation, model + RAM-fit checks, hardware-sized threads/KV/NUMA, launch, and health verification. The host may also contain AMD Instinct GPUs; this skill keeps the endpoint on CPU. Reports and stops on failure; does not debug. ## Owner