Prototype: pin the CPU legs' glibc floor with a container, not the runner - #79
Draft
danielhanchen wants to merge 1 commit into
Draft
Prototype: pin the CPU legs' glibc floor with a container, not the runner#79danielhanchen wants to merge 1 commit into
danielhanchen wants to merge 1 commit into
Conversation
…nner Untested against a real runner. Prototype for review, not to merge. ubuntu-22.04 and ubuntu-22.04-arm retire 2027-04-17. Today those labels are what holds the bundles at glibc 2.35 / GLIBCXX <= 3.4.30 so they load on Ubuntu 22.04 and Debian 12, so the labels cannot simply be swapped: a 24.04 build needs GLIBC_2.38. Moving the userspace into a container decouples the floor from the runner image. The build then keeps 2.35 on a 24.04 host, and the same shape survives the 24.04 retirement after it. The CPU legs are the prototype because they are the only Linux build legs with no vendor SDK to install, and because they take their source from download-artifact rather than checkout, so the container needs no git. Adjustments the bare image forces: no privilege prefix, since the container is root and ships none; and ca-certificates, wget, gnupg and lsb-release, which the host image has and ubuntu:22.04 does not, all of which llvm.sh needs on arm64. Also adds the assertion the runner label was silently standing in for. Nothing ever checked the produced floor, so nothing would have caught the label drifting. objdump now reports the highest GLIBC_x.y across bin/ and fails the job above 2.35, which makes the container's effect checkable rather than assumed. Not converted: CUDA, ROCm and Vulkan. Each installs a vendor toolkit whose behaviour inside a container needs its own test, and the arm64 CUDA leg is already on ubuntu-24.04-arm at glibc 2.39, so its floor is a separate question.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Untested against a real runner. Opened for review of the approach, not to merge.
The problem the label swap cannot solve
ubuntu-22.04andubuntu-22.04-armretire 2027-04-17. Today those labels are what holds the bundles at glibc 2.35 / GLIBCXX <= 3.4.30 so they load on Ubuntu 22.04 and Debian 12. A 24.04 build needsGLIBC_2.38, so the label cannot simply be swapped the way it can for the orchestration jobs.Moving the userspace into a container decouples the floor from the runner image: the build keeps 2.35 on a 24.04 host, and the same shape survives the 24.04 retirement after it.
Why the CPU legs
They are the only Linux build legs with no vendor SDK to install, and they take their source from
download-artifactrather thancheckout, so the container needs no git.What the bare image forces
ubuntu:22.04ships none.ca-certificates,wget,gnupg,lsb-release. Present on the host runner image, absent from the base, andllvm.shneeds all four on arm64.The assertion the runner label was standing in for
Nothing has ever checked the produced floor. It was inherited from the runner image and trusted, which means nothing would have caught the label drifting either.
This adds a step that reads the highest
GLIBC_x.yany binary inbin/asks the loader for and fails above 2.35. That makes the container's effect checkable rather than assumed, and it is arguably worth having on the build legs regardless of which approach wins.What I could not verify
I have no container runtime available, so none of this has run. Specifically unproven:
ccache-action,download-artifact,upload-artifactandcache/savebehave inside the containerllvm.shpath works with only the packages listed aboveThe first real run answers all four at once. Worth doing on a staging repo before this goes near a release.
Not converted
CUDA, ROCm and Vulkan each install a vendor toolkit whose behaviour inside a container needs its own test. Note also that the arm64 CUDA leg already runs on
ubuntu-24.04-armat glibc 2.39, so its floor is a separate question from the x64 profiles rather than the same one.No upstream precedent
ggml-org/llama.cppuses containers for toolchain provisioning, not floor management: their container base always matches or exceeds the runner, and their ROCm jobs still pinubuntu-22.04. So they carry the same exposure and there is no working example of this to copy.