ci(release): publish the operator image + chart, and a promotion manifest - #89
Open
jiejingzhangamd wants to merge 5 commits into
Open
Conversation
jiejingzhangamd
requested review from
JohnQinAMD,
limou102 and
xiaobochen-amd
as code owners
August 5, 2026 07:24
…fest Based on #90, which makes the manager a component tag of the shared repo (rocm/infera:operator-<version>) rather than its own -operator-manager repository. This follows that convention. Two things a release did not produce, both left to a human afterwards. **The operator manager image and its Helm chart.** The build already existed as make docker-build / docker-push / helm-push, but nothing ran it on a tag, so `helm install` pulled whatever manager image happened to be committed in values.yaml. The new job pins Chart.yaml's version and appVersion, and values.yaml's image repository and tag, to this release before packaging -- so a plain `helm install` gets the manager this run built. It runs on a GitHub-hosted runner: the operator Dockerfile is a distroless CGO_ENABLED=0 Go build, needing neither a GPU nor the SLURM dispatch the engine images go through. Deliberately not gated on `build` or `overlay` -- the operator ships no engine code, so waiting on a GPU queue would only delay a chart nobody was blocked on. The chart version drops the leading `v` while the image tag keeps it: Helm requires SemVer and `helm package` rejects "v0.2.3". The two differ by that character alone, which is worth knowing before it looks like a bug. **promotion-<id>.json.** Engine images publish to the private staging repo and are promoted to the public one after review; that promotion is a separate human-gated process, and the source-to-destination list was written by hand each time. It is now generated from `prepare`'s component list, so a partial dispatch produces a partial manifest instead of naming images that were never built. Uploaded as a workflow artifact and, on a tag, attached to the Release. Because #90 makes the manager a component tag of the same repo, it promotes exactly like the engines and is included -- which is what settles the question of whether it belonged in the manifest at all. Also hoists the image-id computation into `prepare`. It was open-coded identically in `build` and `overlay`; harmless while they agreed, but the overlay harvests the engine images *by tag*, so an id that diverged between jobs would silently harvest the wrong ones. Verified locally on this baseline: the manifest generator emits 7 entries (5 engines + overlay + operator) in the expected shape, and the chart-pinning steps render `docker.io/inferaimage/infera:operator-v0.2.3` and package to infera-operator-0.2.3.tgz. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
jiejingzhangamd
changed the base branch from
main
to
feature/weilei/build_operator
August 5, 2026 15:44
jiejingzhangamd
force-pushed
the
release/operator-image-and-promotion
branch
from
August 5, 2026 15:44
4152bc7 to
4ef5651
Compare
The chart and every recipe named things that do not exist. Checked against Docker Hub anonymously, the way a reader would: rocm/infera-operator (chart, 6 recipes) 404 rocm/infera:operator-v0.1.0 (values.yaml) 404 inferaimage/infera-operator-manager:0.1.2 200, publicly pullable So `helm install ... oci://docker.io/rocm/infera-operator` fails outright, and installing the chart *from source* was no better: it installs cleanly and then sits in ImagePullBackOff, which reads as a cluster problem rather than a manifest one. Docker Hub answers 401 rather than 404 for a missing repo when unauthenticated, so the error does not even say the thing is absent. The operator is not part of the tagged release. Engine images publish to staging and are promoted after review; the manager does not go through that, and is published on its own to inferaimage/infera-operator-manager. That is what the chart now defaults to. The recipes install from `deploy/operator/helm/infera-operator` instead of a published chart, because that is what works today. The pinned `--version 0.1.0` goes with it -- even once a chart is published, a hard-coded version in a recipe is stale the moment the next tag lands. Verified end to end on k3s, not just rendered: chart installs, the manager Pod reaches Running with 0 restarts, the CRD arrives with the chart, and a real InferaDeployment reconciles to a 1/1 child Deployment with the CR reporting `ready`. When the release does publish the operator (#89), the chart default and the recipes move to the published artefact together -- and that change should be made after a release has actually produced one, not before. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
Build infera-router and delete the toolchain in the same RUN, unconditionally -- including on bases that shipped cargo themselves (sglang, atom). This is about what an image scanner reports, not about size. Scanners read the flattened final filesystem, so a toolchain deleted in the same RUN is genuinely absent from what gets scanned, while one deleted in a later step is not. A serving image has no business shipping a compiler toolchain regardless of who put it there. An earlier version of this change kept a base-provided toolchain on the grounds that removing it saves no bytes. That optimised for the wrong thing. Verified by building against all three base families and scanning the resulting images from outside the build: vllm (no cargo in base, rustup path) cargo/rustc/rustup gone, 0 residue sglang (cargo 1.97 in base) gone; sglang import + launch_server OK atom (cargo 1.94 in base) gone; infera-router OK Nothing at runtime needs cargo: the engines are Python, and infera-router links only against base system libs (libc/libstdc++/libgcc/libm). Two in-RUN assertions keep this honest -- cargo must be off PATH, and infera-router must still answer --help after the cleanup. Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
jiejingzhangamd
force-pushed
the
release/operator-image-and-promotion
branch
from
August 6, 2026 01:06
2f63fb0 to
514529c
Compare
manual/recipes/kimi-k3.md and glm5.2.md went straight from installing the
operator to `kubectl apply -f .../deploy.yaml`. Every manifest hardcodes
`namespace: infera` and nothing creates it, so the very first apply after the
operator -- model-cache.yaml, before any recipe manifest -- fails with
Error from server (NotFound): namespaces "infera" not found
The four other recipe pages already had the step; these two were the ones a
reader actually lands on from the manual.
Also carry over the KUBECONFIG note from kimi-k3-optimized.md. On k3s, kubectl
finds /etc/rancher/k3s/k3s.yaml implicitly and helm does not, so the operator
install fails with "Kubernetes cluster unreachable: localhost:8080" while every
kubectl command around it works. Hit while verifying this.
Verified on a clean single-node k3s (no infera CRD, no infera namespace):
the Prerequisites block now runs start to finish, the chart installs the
InferaDeployment CRD (timestamp 16 s after cluster creation, so it came from
the chart and not from a prior install), the operator rolls out and acquires
its lease with no RBAC errors, and all 13 recipe manifests pass
`kubectl apply --dry-run=server` against the freshly installed CRD.
All five images the recipes reference resolve without credentials.
Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
The gotcha told readers to install k3s with `--data-dir` on a large disk and
left it there, which reads as "do this and DiskPressure cannot happen". It can.
`--data-dir` moves the image store, so kubelet's imagefs follows it. kubelet's
nodefs does not: its root stays at `/var/lib/kubelet` on the OS disk, and the
default eviction threshold is `nodefs.available<10%` of that disk.
Diagnosed on a node that had done exactly what the doc said -- `--data-dir` on
a 7 TB NVMe -- and still carried the `disk-pressure` taint for two days, having
evicted the operator four times. kubelet's own accounting shows the split:
nodefs cap=837.5G -> / (OS disk, 0 bytes free)
imagefs cap=7152G -> /mnt/k3local (--data-dir target, 40% free)
Freeing the OS disk to 18.6% cleared the condition 20 s later.
Points at `/api/v1/nodes/<node>/proxy/stats/summary` because that is what
distinguishes the two filesystems; `df` on the data-dir looks fine throughout
and sends you looking in the wrong place.
Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things a release did not produce, both left to a human afterwards.
1. The operator image and its Helm chart
The build already existed —
make docker-build/docker-push/helm-pushindeploy/operator— but nothing ran it on a tag. Sohelm installpulledwhatever manager image happened to be committed in
values.yaml, which is how achart quietly installs a months-old manager.
The new job pins, before packaging:
Chart.yamlversion/appVersionvvalues.yamlimage.repository<IMAGE_REPO>(the shared repo)values.yamlimage.tagoperator-<tag>, with thevso a plain
helm installgets the manager this run built. Thevasymmetry isnot a slip: Helm requires SemVer and
helm packagerejectsv0.2.3.It runs on a GitHub-hosted runner — the operator Dockerfile is a distroless
CGO_ENABLED=0Go build, needing neither a GPU nor the SLURM dispatch theengine images go through. Deliberately not gated on
buildoroverlay:the operator ships no engine code, so waiting on a GPU queue would only delay a
chart nobody was blocked on.
Pushing goes through the existing Makefile targets rather than open-coding
helmhere, so there is one definition of how a chart is built and named.2.
promotion-<id>.jsonEngine images publish to the private staging repo and are promoted to the public
one after review. That promotion is a separate, human-gated process — this job
does not perform it. It writes down, for this exact release, which source
maps to which destination:
[ {"source": "inferaimage/infera:sglang-v0.2.3", "destination": "rocm/infera:sglang-v0.2.3"}, {"source": "inferaimage/infera:vllm-v0.2.3", "destination": "rocm/infera:vllm-v0.2.3"}, ... ]Generated from
prepare's component list, so a partial dispatch produces apartial manifest rather than naming images that were never built — which is
the failure mode of writing the list by hand afterwards. Uploaded as a workflow
artifact, and on a tag also attached to the GitHub Release.
Destination defaults to
rocm/infera, overridable with aPROMOTE_TO_REPOActions variable.
Also: one id, not three
The image-id computation was open-coded identically in
buildandoverlay.Harmless while they agree — but the overlay harvests the engine images by
tag, so an id that diverged between jobs would silently harvest the wrong
ones. It is now computed once in
prepareand passed down.Verified locally
v0.2.3tag (6 entries: 5 engines + overlay).
version: 0.2.3,appVersion: "0.2.3",image: docker.io/inferaimage/infera-operator-manager:v0.2.3, render throughhelm template, and package toinfera-operator-0.2.3.tgz.Not verified: the pushes themselves. Neither the manager image, the chart,
nor the release upload has run — that needs a real tag with credentials. The
first release after this merges is the real test, and the failure mode is loud
(a failed job), not silent.
The operator is now in the manifest
My earlier open question — whether the manager belonged in the promotion list —
is settled by #90. Making it a component tag of the same repo means it promotes
exactly like the engines, so it is included:
{"source": "inferaimage/infera:operator-v0.2.3", "destination": "rocm/infera:operator-v0.2.3"}7 entries: 5 engines + overlay + operator.
Still open: does a relayed image stay the same image?
The manifest maps tag to tag, with
sourcederived from wherever the buildpushed. That is exact for a direct copy. It is not exact if the images pass
through an intermediate registry, for two separate reasons:
sourcemay not be where the promoter pulls from. Easy to fix — make thesource repo configurable independently of
IMAGE_REPO.tag; a re-tag or a re-push through a mirror produces a
rocm/infera:...thatis not what was built and reviewed, and the manifest cannot tell. This is
not hypothetical for us: 13 recipes pin the overlay by
@sha256:preciselybecause digests are immutable and tags are not.
The fix, if there is a relay, is to carry the digest:
{"source": "...", "source_digest": "sha256:…", "destination": "..."}which turns the manifest from a copy instruction into something verifiable after
the fact. The digest is free —
docker pushalready prints it.Not doing this yet: it depends on whether there is a relay and where, and on
whether the promotion tool tolerates an extra field or validates a strict
schema. Worth confirming before adding.