prebuilt: make the ROCm ccache usable, and hold one ROCm alpha per week - #88
Conversation
The ROCm legs install a rolling nightly toolchain, but the cache key only carried the gfx target and the source tag. ccache hashes compiler identity into every entry, so a cache written by a different nightly can never hit: the job paid a 64-128 MB restore, missed all 787 objects, then re-saved. Every ROCm cache in the repo has last_accessed_at == created_at. Also hash the compiler by content on Linux. The action already does this on Windows and macOS, but Linux was left on the default mtime check, and ROCm is re-extracted from the tarball on every run.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…ghtly TheRock publishes a dev alpha every day, between about 03:00Z and 09:00Z, and the nightly cron fires at 20:13Z, so every scheduled run picked up a new toolchain and rebuilt ROCm from scratch. The GPU targets we ship are all covered by much older builds, so the daily churn bought nothing. rocm_version now defaults to weekly: take the newest alpha dated on or before the most recent Sunday in SF time. That is a pure function of the date, so it needs no state and no bot, and every run from Sunday to Saturday resolves the same build. Combined with the version-keyed ccache the whole week hits. latest and explicit versions still work.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
danielhanchen
changed the title
…eekly cutoff once The cache pruner groups by the restore prefix, which is the key minus its tag suffix. Putting the ROCm version in the key made every weekly toolchain its own group, so it kept two caches per version forever and none of them could ever be restored again. Strip the version in the pruner too, so all generations of a leg share one group and only the newest two survive. The weekly cutoff was also computed per matrix leg off each leg's own clock, so a run crossing the SF Saturday-to-Sunday boundary could ship some bundles from last week's toolchain and some from this week's. Resolve it once in the parent and pass it down; a standalone call to the child still computes its own. Also unwrap the comment sentences added by this branch, per AGENTS.md.
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
The cutoff was the most recent Sunday, which is the day the first run of the week executes, so the index can still gain a qualifying build after that run has read it. TheRock names a build for the next day and usually publishes the evening before, but on 2026-07-05 the Sunday build landed at 18:33 PT, over five hours after the 13:13 PT cron. That Sunday resolved 20260704 and the rest of the week resolved 20260705, so the week used two toolchains and paid two cold builds, which is what weekly exists to avoid. It is 1 of the last 12 Sundays, and four more cleared the cron by only a few hours. Use the Saturday before instead. It has had at least 17 hours to settle in every observed case, and every day Sun-Sat still maps to one cutoff. Also let the prune regex match a vanilla tag: it required -mix-<hash>, so on a build with an empty pr-set every key became its own group and nothing was ever pruned.
|
Codex Review: Didn't find any major issues. Nice work! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
The ROCm legs were getting a 0% ccache hit rate while CUDA/CPU/Vulkan sit at 90-94%, which made
ROCm the whole critical path:
ROCm / windows/gfx110Xtook 1h10m in run 31277751235 and gatedthe 1h46m publish job.
Two things caused it, and this PR fixes both.
1. The cache key did not include the ROCm version
The ROCm legs install a rolling toolchain from TheRock, but the key carried only the gfx target
and the source tag. ccache hashes compiler identity into every entry, so a cache written by a
different build can never hit. Each job paid a 64-128 MB restore, missed all 787 objects, then
re-saved 128 MB. Every ROCm cache in the repo had
last_accessed_at == created_at, so not onehad ever been read back.
Both jobs now key on the resolved
DETECTED_ROCM_VERSION, which is already in the environmentby the time the ccache step runs. The restore-key prefix stops at the version, so a run no
longer downloads a cache that cannot hit. This is what ggml-org does upstream
(
release.yml:key: release-ubuntu-22.04-rocm-${{ matrix.ROCM_VERSION }}).Linux also now hashes the compiler by content. The pinned action already does this on Windows
and macOS but leaves Linux on the default mtime check, and ROCm is re-extracted from the tarball
every run. Not using
%compiler% --versionon purpose: ROCm's clang reports a generic versionstring that can be identical across different builds, so it could serve an object built by
another compiler.
2. The toolchain moved every single night
TheRock publishes a dev alpha daily between about 03:00Z and 09:00Z; the cron fires at 20:13Z.
So every scheduled run picked up a new compiler and rebuilt ROCm from scratch, and no cache key
scheme could have helped.
The daily churn was not buying anything. Every GPU target we ship is covered by much older
builds: upstream builds the same set (including gfx1151 and gfx1200/1201) on pinned ROCm 7.2.1
and HIP SDK 26.Q1. The nightlies index is 100%
aalpha builds, and the version churn(7.15.0a to 10.1.0a in two weeks) is dev snapshot numbering, not releases. Over the last 20
pipeline runs the ROCm legs were 143 success, 10 skipped, 0 failures, so nothing points at a
nightly fix we depend on.
rocm_versionnow defaults toweekly: the newest alpha dated on or before the most recentSunday in SF time. That is a pure function of the date, so it needs no pin file, no bot and no
state, and every run from Sunday to Saturday resolves the same build.
latestand explicitversions still work.
Replaying the workflow's own jq expression over the last four weeks:
Constant inside each Sun-Sat window, rolls only on Sunday. The PowerShell path gives identical
results. Gaps are handled: today is Sunday 09 Aug but no 0808 or 0809 exists, so it picks 0807.
A stateless pin would only break if a build dated on or before Sunday showed up after Sunday's
run. Only 1 of the last 40 builds was published later than its build date, TheRock never
backfills a skipped day (the 13, 14 and 22 July holes are still holes), and publishes land
before the cron. Worst case is one extra rebuild, never a wrong binary.
Measured
Both runs on this branch with
tag=b10327,gfx_target=gfx110X,operating_systems=ubuntu,publish=false, same toolchain10.1.0a20260807:Output is unchanged. Comparing the two bundles file by file, 1128 of 1129 files are
byte-identical. The one that differs is
libllama-common.so, and that is a source difference,not a caching artifact:
resolvemints a new merge commit per run and bakes its short SHA intobuild-info. ccache missed exactly one object out of 787, which is that translation unit.
As a control, the same single file is also the only difference between run 31297076947 and the
b10327-mix-b1e9972 release built last night, which was itself a full 0%-hit build. Two
independent uncached builds differ exactly as much as cold-vs-warm does, so caching adds no
variance.
Splitting host TUs onto a stable compiler was considered and dropped. Ninja timings from the
cold run: 145 device steps take 20.4m, 645 host steps take 4.6m, link and copy 0.3m. The device
TUs are 80% of the wall clock and are exactly what a toolchain bump invalidates, so the split
would buy little for real ABI risk.
Expected effect: 6 nightlies in 7 hit the cache, ROCm stops being the critical path except on
Sundays.