Skip to content

ci(release): publish the operator image + chart, and a promotion manifest - #89

Open
jiejingzhangamd wants to merge 5 commits into
feature/weilei/build_operatorfrom
release/operator-image-and-promotion
Open

ci(release): publish the operator image + chart, and a promotion manifest#89
jiejingzhangamd wants to merge 5 commits into
feature/weilei/build_operatorfrom
release/operator-image-and-promotion

Conversation

@jiejingzhangamd

@jiejingzhangamd jiejingzhangamd commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Based on #90 — retargeted at feature/weilei/build_operator, so this diff
is only the workflow. #90 makes the manager a component tag of the shared repo
(rocm/infera:operator-<version>) rather than its own -operator-manager
repository; this follows that convention.

Two things a release did not produce, both left to a human afterwards.

1. The operator image and its Helm chart

The build already existed — make docker-build / docker-push / helm-push in
deploy/operator — but nothing ran it on a tag. So helm install pulled
whatever manager image happened to be committed in values.yaml, which is how a
chart quietly installs a months-old manager.

The new job pins, before packaging:

from
Chart.yaml version / appVersion the tag, without the leading v
values.yaml image.repository <IMAGE_REPO> (the shared repo)
values.yaml image.tag operator-<tag>, with the v

so a plain helm install gets the manager this run built. The v asymmetry is
not a slip: Helm requires SemVer and helm package rejects v0.2.3.

It runs on a GitHub-hosted runner — the operator Dockerfile is a distroless
CGO_ENABLED=0 Go build, needing neither a GPU nor the SLURM dispatch the
engine images go through. Deliberately not gated on build or overlay:
the operator ships no engine code, so waiting on a GPU queue would only delay a
chart nobody was blocked on.

Pushing goes through the existing Makefile targets rather than open-coding
helm here, so there is one definition of how a chart is built and named.

2. promotion-<id>.json

Engine images publish to the private staging repo and are promoted to the public
one after review. That promotion is a separate, human-gated process — this job
does not perform it. It writes down, for this exact release, which source
maps to which destination:

[
  {"source": "inferaimage/infera:sglang-v0.2.3",  "destination": "rocm/infera:sglang-v0.2.3"},
  {"source": "inferaimage/infera:vllm-v0.2.3",    "destination": "rocm/infera:vllm-v0.2.3"},
  ...
]

Generated from prepare's component list, so a partial dispatch produces a
partial manifest
rather than naming images that were never built — which is
the failure mode of writing the list by hand afterwards. Uploaded as a workflow
artifact, and on a tag also attached to the GitHub Release.

Destination defaults to rocm/infera, overridable with a PROMOTE_TO_REPO
Actions variable.

Also: one id, not three

The image-id computation was open-coded identically in build and overlay.
Harmless while they agree — but the overlay harvests the engine images by
tag
, so an id that diverged between jobs would silently harvest the wrong
ones. It is now computed once in prepare and passed down.

Verified locally

  • The manifest generator reproduces the expected JSON exactly for a v0.2.3
    tag (6 entries: 5 engines + overlay).
  • The chart-pinning steps produce version: 0.2.3, appVersion: "0.2.3",
    image: docker.io/inferaimage/infera-operator-manager:v0.2.3, render through
    helm template, and package to infera-operator-0.2.3.tgz.
  • Workflow YAML parses.

Not verified: the pushes themselves. Neither the manager image, the chart,
nor the release upload has run — that needs a real tag with credentials. The
first release after this merges is the real test, and the failure mode is loud
(a failed job), not silent.

The operator is now in the manifest

My earlier open question — whether the manager belonged in the promotion list —
is settled by #90. Making it a component tag of the same repo means it promotes
exactly like the engines, so it is included:

{"source": "inferaimage/infera:operator-v0.2.3", "destination": "rocm/infera:operator-v0.2.3"}

7 entries: 5 engines + overlay + operator.

Still open: does a relayed image stay the same image?

The manifest maps tag to tag, with source derived from wherever the build
pushed. That is exact for a direct copy. It is not exact if the images pass
through an intermediate registry, for two separate reasons:

  1. source may not be where the promoter pulls from. Easy to fix — make the
    source repo configurable independently of IMAGE_REPO.
  2. A tag is not an identity. Any hop can put different bytes behind the same
    tag; a re-tag or a re-push through a mirror produces a rocm/infera:... that
    is not what was built and reviewed, and the manifest cannot tell. This is
    not hypothetical for us: 13 recipes pin the overlay by @sha256: precisely
    because digests are immutable and tags are not.

The fix, if there is a relay, is to carry the digest:

{"source": "...", "source_digest": "sha256:…", "destination": "..."}

which turns the manifest from a copy instruction into something verifiable after
the fact. The digest is free — docker push already prints it.

Not doing this yet: it depends on whether there is a relay and where, and on
whether the promotion tool tolerates an extra field or validates a strict
schema. Worth confirming before adding.

…fest

Based on #90, which makes the manager a component tag of the shared repo
(rocm/infera:operator-<version>) rather than its own -operator-manager
repository. This follows that convention.

Two things a release did not produce, both left to a human afterwards.

**The operator manager image and its Helm chart.** The build already
existed as make docker-build / docker-push / helm-push, but nothing ran
it on a tag, so `helm install` pulled whatever manager image happened to
be committed in values.yaml. The new job pins Chart.yaml's version and
appVersion, and values.yaml's image repository and tag, to this release
before packaging -- so a plain `helm install` gets the manager this run
built.

It runs on a GitHub-hosted runner: the operator Dockerfile is a
distroless CGO_ENABLED=0 Go build, needing neither a GPU nor the SLURM
dispatch the engine images go through. Deliberately not gated on `build`
or `overlay` -- the operator ships no engine code, so waiting on a GPU
queue would only delay a chart nobody was blocked on.

The chart version drops the leading `v` while the image tag keeps it:
Helm requires SemVer and `helm package` rejects "v0.2.3". The two differ
by that character alone, which is worth knowing before it looks like a
bug.

**promotion-<id>.json.** Engine images publish to the private staging
repo and are promoted to the public one after review; that promotion is a
separate human-gated process, and the source-to-destination list was
written by hand each time. It is now generated from `prepare`'s component
list, so a partial dispatch produces a partial manifest instead of naming
images that were never built. Uploaded as a workflow artifact and, on a
tag, attached to the Release.

Because #90 makes the manager a component tag of the same repo, it
promotes exactly like the engines and is included -- which is what
settles the question of whether it belonged in the manifest at all.

Also hoists the image-id computation into `prepare`. It was open-coded
identically in `build` and `overlay`; harmless while they agreed, but the
overlay harvests the engine images *by tag*, so an id that diverged
between jobs would silently harvest the wrong ones.

Verified locally on this baseline: the manifest generator emits 7 entries
(5 engines + overlay + operator) in the expected shape, and the
chart-pinning steps render
`docker.io/inferaimage/infera:operator-v0.2.3` and package to
infera-operator-0.2.3.tgz.

Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
@jiejingzhangamd
jiejingzhangamd changed the base branch from main to feature/weilei/build_operator August 5, 2026 15:44
@jiejingzhangamd
jiejingzhangamd force-pushed the release/operator-image-and-promotion branch from 4152bc7 to 4ef5651 Compare August 5, 2026 15:44
The chart and every recipe named things that do not exist. Checked
against Docker Hub anonymously, the way a reader would:

  rocm/infera-operator (chart, 6 recipes)     404
  rocm/infera:operator-v0.1.0 (values.yaml)   404
  inferaimage/infera-operator-manager:0.1.2   200, publicly pullable

So `helm install ... oci://docker.io/rocm/infera-operator` fails
outright, and installing the chart *from source* was no better: it
installs cleanly and then sits in ImagePullBackOff, which reads as a
cluster problem rather than a manifest one. Docker Hub answers 401 rather
than 404 for a missing repo when unauthenticated, so the error does not
even say the thing is absent.

The operator is not part of the tagged release. Engine images publish to
staging and are promoted after review; the manager does not go through
that, and is published on its own to inferaimage/infera-operator-manager.
That is what the chart now defaults to.

The recipes install from `deploy/operator/helm/infera-operator` instead
of a published chart, because that is what works today. The pinned
`--version 0.1.0` goes with it -- even once a chart is published, a
hard-coded version in a recipe is stale the moment the next tag lands.

Verified end to end on k3s, not just rendered: chart installs, the
manager Pod reaches Running with 0 restarts, the CRD arrives with the
chart, and a real InferaDeployment reconciles to a 1/1 child Deployment
with the CR reporting `ready`.

When the release does publish the operator (#89), the chart default and
the recipes move to the published artefact together -- and that change
should be made after a release has actually produced one, not before.

Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
Build infera-router and delete the toolchain in the same RUN, unconditionally
-- including on bases that shipped cargo themselves (sglang, atom).

This is about what an image scanner reports, not about size. Scanners read the
flattened final filesystem, so a toolchain deleted in the same RUN is genuinely
absent from what gets scanned, while one deleted in a later step is not. A
serving image has no business shipping a compiler toolchain regardless of who
put it there.

An earlier version of this change kept a base-provided toolchain on the grounds
that removing it saves no bytes. That optimised for the wrong thing.

Verified by building against all three base families and scanning the resulting
images from outside the build:

  vllm  (no cargo in base, rustup path)  cargo/rustc/rustup gone, 0 residue
  sglang (cargo 1.97 in base)            gone; sglang import + launch_server OK
  atom   (cargo 1.94 in base)            gone; infera-router OK

Nothing at runtime needs cargo: the engines are Python, and infera-router links
only against base system libs (libc/libstdc++/libgcc/libm).

Two in-RUN assertions keep this honest -- cargo must be off PATH, and
infera-router must still answer --help after the cleanup.

Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
@jiejingzhangamd
jiejingzhangamd force-pushed the release/operator-image-and-promotion branch from 2f63fb0 to 514529c Compare August 6, 2026 01:06
manual/recipes/kimi-k3.md and glm5.2.md went straight from installing the
operator to `kubectl apply -f .../deploy.yaml`. Every manifest hardcodes
`namespace: infera` and nothing creates it, so the very first apply after the
operator -- model-cache.yaml, before any recipe manifest -- fails with

    Error from server (NotFound): namespaces "infera" not found

The four other recipe pages already had the step; these two were the ones a
reader actually lands on from the manual.

Also carry over the KUBECONFIG note from kimi-k3-optimized.md. On k3s, kubectl
finds /etc/rancher/k3s/k3s.yaml implicitly and helm does not, so the operator
install fails with "Kubernetes cluster unreachable: localhost:8080" while every
kubectl command around it works. Hit while verifying this.

Verified on a clean single-node k3s (no infera CRD, no infera namespace):
the Prerequisites block now runs start to finish, the chart installs the
InferaDeployment CRD (timestamp 16 s after cluster creation, so it came from
the chart and not from a prior install), the operator rolls out and acquires
its lease with no RBAC errors, and all 13 recipe manifests pass
`kubectl apply --dry-run=server` against the freshly installed CRD.
All five images the recipes reference resolve without credentials.

Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
The gotcha told readers to install k3s with `--data-dir` on a large disk and
left it there, which reads as "do this and DiskPressure cannot happen". It can.

`--data-dir` moves the image store, so kubelet's imagefs follows it. kubelet's
nodefs does not: its root stays at `/var/lib/kubelet` on the OS disk, and the
default eviction threshold is `nodefs.available<10%` of that disk.

Diagnosed on a node that had done exactly what the doc said -- `--data-dir` on
a 7 TB NVMe -- and still carried the `disk-pressure` taint for two days, having
evicted the operator four times. kubelet's own accounting shows the split:

    nodefs   cap=837.5G  ->  /             (OS disk, 0 bytes free)
    imagefs  cap=7152G   ->  /mnt/k3local  (--data-dir target, 40% free)

Freeing the OS disk to 18.6% cleared the condition 20 s later.

Points at `/api/v1/nodes/<node>/proxy/stats/summary` because that is what
distinguishes the two filesystems; `df` on the data-dir looks fine throughout
and sends you looking in the wrong place.

Signed-off-by: Zhang, Jiejing <jiejing.zhang@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant