Metal: platform-aware expert-cache defaults via honest storage probe (#379) - #386
Conversation
|
Heads-up: |
f3a7fb9 to
0822b18
Compare
|
Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic Rebase after #391This PR was rebased (relocation-only, no behavior change) from engine base ProblemOn Apple Silicon (Metal backend, unified memory, fast NVMe), colibri's stock
Scope caveat, to prevent over-reading: the tuned row is a four-knob Issues #180 (M3 Ultra) and #107 (M4 Pro) corroborate the same shape on other Mechanism (#379)Metal's per-buffer residency cost makes the engine's own expert LRU cache Whether a volume is "fast" has to be measured, not assumed: buffered DesignS1 -- honest storage probe ( S2 -- platform-aware cache defaults (
MTP, RAM sizing, and autopin are untouched -- out of scope for this PR. Precedence (
An explicit setting is never overridden by the probe, at any tier. Note on the cap/CAP_RAISE coupling (deliberate): on a qualifying box, an S3 -- the one-line notice ( Silent when the probe ran but storage was slow -- defaults stay at their S4 -- doctor/plan surfacing ( What's explicitly out of scope
Behavioral disclosures (small, but stated plainly)
Durable vs. current-state (a review map)Two kinds of change ship here, and they age differently: Durable machinery -- the F_NOCACHE probe and Current-state calibration -- the qualifying-platform values: cap 1
In both cases the edit is confined to the platform-default branch of TestingAuthor-side testing was parse-level and unit-level only -- no model was
Post-rebase gates (performed against On-box validation (performed 2026-07-18, predates this rebase): M5 Max 128
These numbers predate the rebase onto Post-rebase on-box verification (2026-07-19, M5 Max 128 GB, macOS 26.5, base
|
monotophic
changed the title
|
Rebase needed: this went stale against |
0822b18 to
57b0b76
Compare
monotophic
changed the title
|
Re-based to dev 68ac9ff and verified clean at my end. |
|
Rebase heads-up: |
57b0b76 to
758ceb2
Compare
|
Rebased, requested verification in my queue. Should be available and posted tonight or tomorrow AM. Machine is pretty tied up and hot at the moment. |
|
Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic. Rebase: head The before/after measurement (M5 Max 128 GB, macOS 26.5, plugged; interleaved
Two observations:
Let me know if you have questions or if there is anything else I can do. |
monotophic
changed the title
758ceb2 to
12c8209
Compare
|
This conflicts with The substance of this one — platform-aware Metal cache defaults driven by an honest storage probe rather than a guess — is something I want, and it's the same discipline as your One thing worth confirming while you're in there: #457 (your grouped-int4 Metal GEMV) and this PR both touch Metal defaults. If they interact — e.g. if fmt=4 support changes what a sensible default cache size is — say which should land first and I'll sequence them that way rather than discovering it at merge time. |
12c8209 to
3e6c086
Compare
|
Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic Rebased: r6, head Same discipline as r4/r5: Gates on the new base (M5 Max, macOS 26.5):
Sequencing: full four-PR order + reasoning is on #457 (short version: |
|
This conflicts with |
3e6c086 to
b68cdf7
Compare
|
Rebased and ready for review. Verification on b68cdf7: git range-diff against r6 shows commits 1 and 3 patch-identical; commit 2 differs only in the Makefile TEST_BINS line (the disclosed superset — dev's test_router_nan + this PR's test_cap_precedence). All three CURRENT-STATE CALIBRATION markers are present with their measurement coordinates intact (2026-07, macOS 26.5, M5 Max). Gates on this head: make check (full C suite including this PR's own test_cap_precedence, plus the Python suite with its probe/cap tests — 13 platform skips), METAL=1 build with zero warnings, and metal-test under both COLI_METAL_RESSET states. Behavior evidence (notice fires, probe caches to .coli_ssd, explicit --cap honored, cached-probe reuse) carries from the r4–r6 validation runs — the rebase is patch-identical at those code paths, and we'll re-run the smoke on request. |
b68cdf7 to
4f7006b
Compare
… S4)
`coli doctor` and `coli plan` now read <model>/.coli_ssd, if the
engine has already written it, and display the measured GB/s. Purely
read-and-display: never re-measures, never guesses, and shows nothing
before the first Metal+darwin engine startup.
resource_plan.read_ssd_probe() is the shared reader; build_plan()
threads it into plan["ssd_probe_gbs"] (None if not yet cached),
format_plan() adds one line when present, and doctor.run_doctor() adds
a storage.ssd_probe check ("pass" with the cached value, or "skip"
before the first probe / when the model itself is invalid).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… cache (JustVugg#379) F_NOCACHE bypasses the page cache only for pages that are not already resident: probing a warm shard measured 22.1 GB/s of RAM and cached it as disk bandwidth, permanently latching the fast-SSD cap default on hardware that never earned it. The probe now snapshots residency with mincore and reads only 4 MB windows whose every page is cold; when the file offers fewer than 64 MB of such windows the measurement is vetoed as contaminated -- nothing is cached, one stderr line says why, and the probe retries on the next (colder) startup. The .coli_ssd cache gains a v2 format, "v2 <gbs> <st_dev>", under a strict grammar (digits[.digits], 0 < gbs < 1000, no strtod permissiveness): a cache is trusted only when it parses AND its st_dev matches the model dir's current volume, so a dir rsync'd to another drive re-probes instead of carrying the old hardware's number. Legacy bare-number caches (possibly warm-contaminated) re-probe once and upgrade. The cache write goes through mkstemp+rename: unique name, so a pre-planted symlink at a predictable temp path can no longer redirect it. The decision layer (grammar parser, cold-tile steering, veto) is pure C, pinned by tests/test_ssd_probe.c over injected residency vectors and the shared grammar-vector file (tests/fixtures/ssd_cache_vectors.txt, chewed by the Python reader's tests too). The probe block now compiles only for its consumer (COLI_METAL on __APPLE__, or the test define), removing the default build's coli_ssd_probe_cached orphan.
…ustVugg#379) The Python reader mirrors the engine's new strict .coli_ssd contract byte-for-byte: parse_ssd_cache() classifies raw bytes under exactly the grammar colibri.c's coli_ssd_cache_parse() accepts (both suites consume the same vector file, tests/fixtures/ssd_cache_vectors.txt, so the two readers cannot drift), and read_ssd_probe() surfaces a value only when a v2 cache's recorded st_dev still matches the model dir's volume. A legacy bare number (possibly measured through a warm page cache) or a cache rsync'd to another drive is one the engine will re-measure -- showing it in doctor/plan would report a number the engine no longer believes. float() permissiveness is gone with it: "inf" used to sail through and reach doctor's JSON as the bare literal Infinity, which json.loads rejects; the new doctor test pins that a corrupt cache can never break machine consumers of --json again.
…nes (JustVugg#379) The 0=auto cap sentinel is a colibri.c contract: coli_resolve_cap resolves it platform-aware. inkling.c reads the same argv as "fit the expert LRU to all available RAM" (cap <= 0), so forwarding the wrapper's 0 default to a non-glm engine silently changed its memory behavior from the 8 slots coli always forced before. cap_for_engine() now translates at the one funnel every launch passes through -- Engine's argv -- keyed on the engine binary's identity (COLI_ENGINE can route any arch to the glm binary, and it is the binary that interprets the argv): a non-glm engine receives the legacy 8 when the user gave no explicit cap, an explicit --cap N passes through to any engine, and glm keeps the 0 sentinel. This shim is CURRENT-STATE CALIBRATION, not durable core: it internalizes an inconsistency that lives across the engines (0 = platform-auto in colibri.c, RAM-auto in inkling.c, historically forced 8 in coli). If cap sentinel semantics are ever unified across engines, remove it and re-derive. coli chat's local-server path now also forwards --cap, which it previously dropped on the floor for non-glm models.
…ustVugg#379) Document what a user meets when the F_NOCACHE storage probe leaves its cache in the model dir: the cold-range steering and the contamination veto (why a warm start can print a deferred-probe line), the strict one-line v2 format both readers share, the st_dev volume-identity rule (rsync the dir to another drive and the engine re-probes instead of inheriting the old number), the one-time legacy upgrade, and that the probe measures the primary model dir only under split/mirror layouts.
…st shard (JustVugg#379) A sparse or still-downloading shard defeats the cold-range steering from the other side: its never-resident pages are HOLES, and pread of a hole is VFS zero-fill at RAM speed -- a 10 GB shard with 100 MB of real data measured 73 GB/s through the whole probe path and latched it as disk bandwidth, a fast-lie that survives the download completing. st_blocks*512 must now cover st_size within 12.5% slack before anything is read; an under-allocated shard is contaminated-class -- nothing cached, honest stderr, re-checked next start. Same conservative polarity as the residency veto. The deferral line names the condition that actually fired: not-fully- allocated (finish the download), under-64MB-of-windows (this shard can never host a trustworthy probe -- previously a fully-cold 30 MB shard was told it was "page-cache resident" forever), or genuinely resident (wait for a cold start). The shard picker takes the LARGEST .safetensors instead of readdir-first, so a stray small shard cannot starve the probe below the floor while a full-size neighbor sits untouched. Also: an implausible measurement (0, >=1000 GB/s, non-finite -- numbers the cache grammar itself refuses) no longer counts as fast for the current run; a cache-write failure (read-only model dir) gets one stderr diagnostic instead of silence; the cache file honors the umask instead of forcing 0644; and the tile modulo is explicitly guarded against ntiles==0 rather than relying on the floor to imply it. test_ssd_probe.c grows the matching arms, including a pread recorder that pins the steering INTO probe_raw: a probe that computes tiles[] and then reads random offsets anyway now fails 113 assertions instead of zero.
…eans absent (JustVugg#379) Basename keying broke both ways. COLI_ENGINE users package the glm engine under arbitrary names (glm52, colibri-1.2, glm-metal) -- every one of them was handed the legacy 8 as if explicit, silently disabling the JustVugg#379 platform-auto for exactly the custom-packaging users COLI_ENGINE exists for; and an inkling binary someone names glm got the 0 sentinel back, the leak the shim exists to stop. cap_for_arch() now keys on the MODEL's config.json model_type, end-to-end the same rule coli's model_arch uses (openai_server's main() now shares the helper instead of duplicating it). The sentinel itself becomes honest: --cap defaults to None (flag absent) in both coli and openai_server, and Engine's cap parameter follows. Absent -> glm arch gets the 0 sentinel to resolve platform-aware, non-glm gets the legacy 8. An EXPLICIT --cap N passes through verbatim to any engine for ALL N -- including --cap 0, which for inkling means upstream's RAM-auto again: people who ask for upstream semantics get them. The glm direct-spawn paths (run, chat) translate None to the 0 sentinel at the argv. CapSentinelShimTest now walks the full matrix (arch x arbitrary engine name x cap absent/5/0) against real temp model dirs, and coli chat's --cap ride-along -- kept deliberately; the historical silent eating was the bug, and this is a disclosed behavior change -- gets its own test driving the real cmd_chat with the process boundary faked.
… yet" (JustVugg#379) "no cached probe yet" was a lie whenever a .coli_ssd file sat right there: a legacy cache now reads "legacy cache pending engine upgrade", a v2 from another volume "cache from another volume; the engine will re-probe here", an unparsable file "unreadable cache". ssd_probe_state() classifies (ok / legacy / foreign / garbage / absent), read_ssd_probe() stays the trusted-value view on top of it, the plan carries the state alongside the gbs, and doctor + format_plan share one wording table. The reader is also bounded now: 65 bytes read instead of the whole file -- the strict grammar caps a well-formed cache at 64 bytes, so byte 65 alone already convicts an impostor file of any size.
…asons (JustVugg#379) st_dev can be recycled across sequentially attached external volumes (demonstrated on macOS), so a carried cache can be wrongly trusted on a different drive that inherits the old device id -- say so plainly, with the delete-the-file remedy, and name volume-UUID identity as the follow-up. The veto bullet gains the two new refusal conditions (under-allocated shard, shard too small for 64 MB of windows) and the largest-shard selection.
…'s precondition (JustVugg#379) Two validator-reported gaps, tests only. The picker had no pin: reverting it to readdir-first-match slipped the whole suite. A dir with a below- floor shard alongside a full-size one now asserts pick_shard returns the largest AND that the probe measures where a first-match picker vetoes SMALL. And the sparse e2e arm's skip-precondition called the very gate function under test, so mutating the gate self-disabled the arm; the precondition now reads st_blocks raw from stat (fixture-specific bound: under half allocated), independent of the code it guards -- a neutered gate now fails all four e2e assertions plus the new sparse-beats-small precedence pin (allocation is checked first: "finish the download" is the actionable advice when both conditions hold).
211cdf5 to
543a8e6
Compare
…d vectors (JustVugg#379) The new python tests wrote their .coli_ssd fixtures through Path.write_text, and on Windows text mode translates \n to \r\n -- the strict reader then (correctly) classified every fixture as garbage: six MinGW CI failures, zero on macOS/Linux. The reader and the grammar are right; the fixtures were lying about their bytes. Every fixture write in test_resource_plan.py and test_doctor.py is now write_bytes. Today's lesson is pinned as explicit intent: the shared C/Python vector file gains "v2 14.322 123\r\n" -> garbage, so both parsers stay honest about \r forever (a reader loosened to strip it fails its vector suite on every platform). For the same definitional cleanliness the C cache read and the C-test fixture writers open in binary mode -- Apple-gated today, but the format is exact bytes and a text-mode file API is a latent CRLF source if the gate ever widens.
|
Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic @JustVugg, this is rebased onto current (If you saw a brief red Windows check on an earlier head What our review found (measured on the shipped binaries, not inferred)
What the fixes do (commits 4–9)
⚠ Engine-default inconsistency, internalized — please read
That shim is a current-state calibration, not durable design (marked as Disclosed behavior changes beyond the rebase
Verification (all on
|
| gate | result |
|---|---|
make check |
OK — 255 python tests (242 at base: +13 net), 13 skipped; all C bins incl. the new test_ssd_probe |
behavioral battery (real binary, synthetic shards, mincore-verified states) |
resident → refused+no-latch · cold → measures (n=5: 8.88–8.98) + v2 + trusted on rerun · sparse → refused before any read · volume move → re-probe · legacy/inf/hex/negative/garbage → never trusted, re-measured |
launcher matrix (real coli serve, 18 cells) |
arch {glm,inkling,kimi} × COLI_ENGINE variants × {absent, --cap 5, --cap 0} — every cell as specified above |
| mutation coverage | 25 distinct mutations across three independent reviews, 24 caught by the suite (the 25th was an equivalent mutant); includes reverting each fix to reproduce its measured failure (resident-latch 22.1, sparse-latch 65.9–75.9, sentinel leak) |
| builds | METAL=1 zero warnings; per-commit builds clean across all 10; -Wunused-function census identical to c9ab660 baseline (the round-1 orphan we introduced was fixed in round 2) |
metal-test |
ok under both COLI_METAL_RESSET states |
Capstones, one per load-bearing claim: contamination (resident 22.1 vs cold
8.9, same file, same machine) · sparse (73 GB/s on 99%-holes) · sentinel leak
(launcher argv 8→0 A/B across the base commits) · fix efficacy (each
reverted fix fails its test loudly and reproduces the measured lie).
Notes and current-state calibrations (coordinates: 543a8e6, APFS/M5 Max)
- The 4.0 GB/s threshold is UNCHANGED — we tested the hypothesis that cold
M5 Max readings fall below it and REFUTED it (five verified-cold runs,
8.88–8.98). st_devrecycling and the 12.5 % allocation slack are measured on macOS/
APFS; eagerly-allocating filesystems skip the sparse arm by construction.- Explicit
--cap 0passes to glm as positional 0, which remains glm's own
auto sentinel (argv-level verbatim; pre-existing glm semantics). - A model dir with a broken/missing
config.jsondefaults to arch=glm
(pre-existing routing); the sentinel could then reach a manually-specified
non-glm binary viaCOLI_ENGINE. Unreachable through coli's own routing. - Probe cost when no valid cache exists: one
mincorepass (~17 ms/GB of
shard) + ≤0.35 s of reads, first start only on healthy setups.
If the added scope is unwelcome in this PR, we can split commits 4–10 into a
follow-up PR and land the pure rebase first — but we did not want to hand over
a head containing defects we had already measured.
|
@JustVugg dude, that "needs rebase" tag is hurting my feelings ;) Thanks for all you are putting into this project! Yesterday's merges were huge. As of my circle back this PR is unconflicted and all green checks, ready to merge. This package moved a little because we took a full pass at our (accumulating) validation process and found some issues that are now fixed. As stated in the rebase comment --cap=n behavior is not consistent across the engines and this is not sane/durable behavior. The engine-forks approach makes sense, but you need some infrastructure for enforcing interface-layer conformance so this doesn't just turn into goo (#700 gets this right for usage records; flag semantics need the same treatment). I am ready to move forward with the MacOS CI and also have ideas for how to implement the engine interface-layer conformance verification (it would generalize the same mechanism as the test battery in this PR). Confirm and I will move forward with both or either. Regards, |
…the wildcard Resolved by a maintainer rather than asking for an eleventh rebase, because the conflict was caused by a merge made minutes earlier and the resolution deletes work rather than adding any. The only conflicting hunk was TEST_BINS. This branch appended test_cap_precedence and test_ssd_probe to the hand-written list; dev (JustVugg#731) now derives that list with a wildcard, specifically so that adding a test stops conflicting with every other PR that adds a test. So the branch's append is dropped: both tests are discovered automatically, and their build rules — which merged cleanly, in a different part of the file — are untouched. Verified: TEST_BINS goes from 29 to 31 entries, both new gates included; `make test-c` passes; test_cap_precedence and test_ssd_probe both report ok. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nflicts TEST_BINS was one long shared line listing 28 gates. Every PR that adds a test appended to that same line, so any two such PRs conflicted in the Makefile by construction — even when they touched entirely unrelated parts of the engine. That is not a hypothetical. c/Makefile appears in 26 of the 40 currently open PRs, and JustVugg#386 hit this specific conflict twice across its rebases: "One conflict (both sides appended a test binary to TEST_BINS)" Adding a gate now means adding your tests/test_*.c and its own build rule, which land in different places in the file. Nobody edits a line somebody else is also editing. TEST_EXCLUDE keeps the four test_*.c that were never default gates, plus test_uring which is appended conditionally on Linux just below — a bare wildcard would have promoted it to every platform. Verified the derived set is byte-identical to the hand-written one (29 entries including test_uring on Linux, same names), `make test-c` passes, and a freshly created tests/test_*.c is picked up without touching this line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merged. Thirteen days and eleven rebases after you opened it — the last one being my fault twice over, so let me put on the record what happened rather than just thanking you. I merged #731 an hour ago: it replaces the hand-written
Then I merged that fix before this PR instead of after, and conflicted you an eleventh time — in the very file I had just changed to stop doing that. So I resolved it myself rather than asking you again. Nothing of yours was rewritten. Your commits are intact; the only thing I removed is a line that no longer needs to exist. On the
It was measuring our merge order, not your work. Two things change from here:
You have three more open here — #528 (fp8-e4m3-b128), #529 (the container stamp), and #379/#387 on the issue side. #528 and #529 are both Thank you for the patience, and for the discipline in every one of those rebases — |
2 participants
Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic
Problem
On Apple Silicon (Metal backend, unified memory, fast NVMe), colibri's stock
defaults leave a large performance win on the table. Measured campaign
(issue #379, M5 Max 128 GB):
MTP=0 CAP_RAISE=0 --ram 90 --cap 1)Scope caveat, to prevent over-reading: the tuned row is a four-knob
configuration. This PR changes the defaults of exactly two of those knobs
(cap, CAP_RAISE); MTP and RAM sizing remain manual and out of scope. Stock
Metal runs with these defaults should therefore land somewhere between 0.4
and 2.0 tok/s, not at 2.0 — this PR is a partial capture of the measured
configuration, and how much of the 5x the two cache knobs alone recover is
exactly what the on-box validation run (below) will measure.
Issues #180 (M3 Ultra) and #107 (M4 Pro) corroborate the same shape on other
Apple Silicon boxes. As the maintainer noted in #379: no one maintaining this
repo has Metal hardware, and this PR was explicitly invited from the field.
Mechanism (#379)
Metal's per-buffer residency cost makes the engine's own expert LRU cache
anti-productive on fast NVMe. The OS page cache already streams cold
experts back in cheaper than Metal can re-pin them into GPU-visible buffers,
so a large expert cache just adds residency churn on top of I/O that was
already going to happen. On this class of hardware, a minimal cache (so the
engine gets out of the page cache's way) beats a large one.
Whether a volume is "fast" has to be measured, not assumed: buffered
reads lie (page-cache hits read 23-97 GB/s on the reference box; a real
F_NOCACHEread measured 14 GB/s on the same file). That's why this PR addsan honest storage probe rather than just flipping the default unconditionally
on every Metal build.
Design
S1 -- honest storage probe (
glm.c,coli_ssd_probe_raw/coli_ssd_probe_cached)C-side, in the engine's own startup path, so it covers
glmdirect users andthe
coliwrapper alike -- Python-only logic would miss direct users.Mirrors
compat_open_direct()/iobench.c's existing__APPLE__branch: opena real model shard (readdir order -- any shard is representative for a
bandwidth probe),
fcntl(F_NOCACHE,1), disable readahead, random 16K-alignedpread, single thread, ~0.35s wall-clock budget. The result is cached in<model>/.coli_ssd(plain-text GB/s) so every startup after the first readsa file instead of re-probing; a missing or unparsable cache file just
re-probes. The cache write is atomic (tmp+rename, the same model-dir
convention as
stats_dump_q); if two engines race on a virgin dir (e.g. aconcurrent serve + run pair), both probe under mutual contention and may
cache a low reading -- the safe direction (defaults simply stay historic);
delete
.coli_ssdto re-measure, and an unparsable file re-probesautomatically on the next start.
S2 -- platform-aware cache defaults (
glm.c,coli,openai_server.py)When
g_metal_enabled && darwin && probe-fast:CAP_RAISEdefaults to 0 (previously 1) -- raising the cache back upunder a generous RAM budget is the same anti-pattern the cap default exists
to avoid.
MTP, RAM sizing, and autopin are untouched -- out of scope for this PR.
Precedence (
coli_resolve_cap(), pure function, unit-tested in isolation):--cap NCLICAPenvCAP_RAISEenvcoliwrapper ("0 = auto" sentinel), 64 for a bare./glmwith no positional argumentAn explicit setting is never overridden by the probe, at any tier.
Note on the cap/CAP_RAISE coupling (deliberate): on a qualifying box, an
explicit
--cap 16does not re-enable auto-raise --CAP_RAISEstilldefaults to 0 there. Rationale: the Metal residency cost scales with cache
growth, so auto-raise is the same mechanism the minimal-cache default avoids;
a user who explicitly sizes the cache gets exactly that size. Explicit
CAP_RAISE=1re-enables auto-raise.S3 -- the one-line notice (
glm.c, stderr, alongside the existing config banner)Printed only when the platform cap default actually engages:
Silent when the probe ran but storage was slow -- defaults stay at their
historic values with no extra output.
S4 -- doctor/plan surfacing (
doctor.py,resource_plan.py)coli doctorandcoli planread<model>/.coli_ssdif present and displayit. Read-and-display only: neither command re-measures or guesses. A new
storage.ssd_probedoctor check reports "pass" with the cached GB/s, or"skip" before the first Metal+darwin startup.
What's explicitly out of scope
--ram), autopin -- their defaults are untouched (and arethe other half of the measured 5x; see the scope caveat at the top).
CACHE_ROUTE, sampling -- nothing here can change a generatedtoken. Cache sizing is a pure performance knob: it changes how often an
expert is re-read from disk, never which expert gets computed for a given
token.
(
g_metal_enabledstays 0, or the measured GB/s is under the threshold,default 4.0 GB/s,
COLI_SSD_FAST_GBSto override), and every invocationresolves byte-identically to today -- including the bare
./glmfallbackof 64.
Behavioral disclosures (small, but stated plainly)
--cap 0/CAP=0now mean "auto" on every platform, not just Metal.This is the sentinel that lets the wrapper say "not explicitly set."
Previously a literal cap=0 was accepted and produced a working zero-slot,
always-cold-load cache mode; that mode was undocumented and unused
repo-wide (no test, tool, doc, or wrapper passes 0). Anyone who somehow
relied on it can get the nearest supported equivalent with
--cap 1.coli's--capargparse default changed from8to0(auto).Without this,
colialways forced an explicit8onto the engine commandline, which (per the precedence table) would have permanently shadowed the
platform default for every wrapped invocation.
openai_server.py's--capflag and its internalEngine/serve()defaults got the samesentinel so programmatic callers don't silently bypass auto.
tools/eval_glm.py) keeps its fixed--cap 64deliberately -- benchmarks need a reproducible cache size, not a
platform-dependent auto (commented in the source).
Durable vs. current-state (a review map)
Two kinds of change ship here, and they age differently:
Durable machinery -- the F_NOCACHE probe and
.coli_ssdcache, thecoli_resolve_cap()precedence chain and cap sentinel, the noticemechanism, and the doctor/plan surfacing. None of these encode an opinion
about how big the cache should be; they survive any change in residency
economics.
Current-state calibration -- the qualifying-platform values: cap 1
and CAP_RAISE 0. These encode today's measured economics (per-buffer
Metal residency cost makes a growing expert cache anti-productive, #379)
and are deliberately concentrated so that revisiting them is a two-constant
edit. "Today" is a checkable coordinate, not rhetoric: measured 2026-07 on
macOS 26.5 (M5 Max, 128 GB) against engine base
caa49f7(dev) -- thein-code marker comments carry the same stamp. A reader far from those
coordinates (newer macOS Metal runtime, or an engine that has absorbed the
#379 residency work) should treat the constants as suspect and re-run the
#379 A/B (cap 1 vs cap 16, hit rate vs tok/s) before trusting them. Two
foreseeable developments would flip them:
already flips the sign of caching on the reference box (cap16 ~ +5% over
cap1 with heap-backed slabs); a follow-up PR is planned from the same
investigation. If it lands, the right qualifying default likely becomes a
larger, RAM-scaled cap -- not 1.
runtime-managed residency in current and upcoming macOS releases). If
expert storage migrates there, the per-buffer registration cost -- the
entire reason cap 1 wins today -- changes wholesale.
In both cases the edit is confined to the platform-default branch of
coli_resolve_cap()and the singleCAP_RAISEdefault expression (bothcarry a marker comment in the code); probe, precedence, notice, and
surfacing are unaffected. The
COLI_SSD_FAST_GBSthreshold (4.0) islikewise a current-economics number, with the env override as escape hatch.
Testing
Author-side testing was parse-level and unit-level only -- no model was
loaded and no inference was run:
tests/test_cap_precedence.c-- exhaustive precedence-table coverage ofcoli_resolve_cap(), including bare-invocation vs wrapper-sentinel vsexplicit CLI/env on fast and slow platforms. Portable (no Metal/Apple
hardware required), part of the default
make test-cgate.tests/test_resource_plan.py/tests/test_doctor.py-- cache-fileparsing (
read_ssd_probe: missing, empty, malformed, negative, valid) andthe doctor/plan surfacing contract.
make glm METAL=1and plainmake glmboth compile clean(
-Wall -Wextra); fullmake test-c+ python unittest suites pass.dummy
*.safetensorsfile (noconfig.json, so every run exits beforemodel_init-- deliberately): notice printed +cap=1on a fast volume;.coli_ssdwritten once (atomically) and reused on the next start;--cap 16/CAP=4override silently; forced-slow threshold andnon-Metal builds stay silent at historic defaults; corrupted
.coli_ssdre-probes; bare
./glmstill resolves cap 64.On-box validation (performed 2026-07-18): M5 Max 128 GB, macOS 26.5.2,
real GLM-5.2 int4 model, engine at this branch's HEAD (base
caa49f7,dev). Three-run block, fixed prompt,
temp 0, 32 decoded tokens each:
(
METAL: fast SSD (9.7 GB/s) -- page cache favored, expert cache minimal (cap 1); override with --cap), cap resolved to 1,.coli_ssdwrittenonce. Decode 1.48 tok/s vs 0.4 at the old defaults -- the two cache
knobs alone recover roughly 3.7x of the measured 5x (MTP and RAM sizing
remain manual, as scoped above).
--cap 16: silent (no notice), cap honored at 16/layer,.coli_ssduntouched. Decode 0.86 tok/s despite the hit rate rising24% -> 50% -- a same-session replication of the Metal — growing the expert cache makes decode SLOWER #379 mechanism this PR
encodes: growing the Metal-registered cache makes decode slower even as
it hits more.
.coli_ssdmtime unchanged,same 9.7 GB/s in the notice), cap=1 again, decode 1.49 tok/s
replicating run 1; temp-0 output text identical across the two stock runs.