feat: Add NUMA-aware RAM-disk streaming

[BColsey]

Summary

• add a Linux-only, Python standard-library coli ramdisk TUI and equivalent scriptable plan, prepare, status, benchmark, start, stop, and destroy commands
• discover NUMA/kernel capabilities, calculate exact staging and runtime reserves, mount THP-enabled tmpfs safely, and stage full models or profile-guided shard closures
• add descriptor-verified tmpfs direct mapping for complete experts while preserving the existing buffered, O_DIRECT, pipeline, and io_uring paths for SSD fallback
• support interleaved and per-node managed engines with durable KV/usage state, crash-safe usage merging, process/mount identity checks, benchmarking, and cleanup
• package and document the new control-plane module and add focused lifecycle, CLI, engine, integration, and packaging tests

Motivation

Large MoE models are frequently limited by repeated expert reads from SSD. This adds a managed tmpfs tier that can stage all weights, or only profile-selected shard closures, while letting eligible experts bypass slab allocation and LRU I/O through read-only mappings. Unstaged experts continue to use Colibri's existing SSD streaming path.

The canonical model is never modified. Managed state stays on durable storage, and the implementation does not change global swap settings or HugeTLB reservations.

Safety and compatibility

• sudo is limited to the exact mount and unmount operations
• mounts, PIDs, process groups, state paths, fingerprints, and safetensors headers are verified before destructive or signaling operations
• preparation rechecks memory reserve before every copy and uses atomic publication
• COLI_RAMMAP=1 and legacy COLI_MMAP=1 are mutually exclusive
• behavior is unchanged when the new RAM-disk variables are unset

Validation

• make check — portable build, full C suite, and 141 Python tests passed with zero compiler warnings; one privileged integration remains opt-in in the default suite
• isolated real-tmpfs prepare/status/destroy integration passed, including no swap growth and durable KV preservation
• git diff --check origin/dev...HEAD passed
• rebased onto current origin/dev; upstream storage-read, sampling, filesystem-detection, Windows, Nix, and documentation changes remain covered by their tests

Hardware follow-up

Full 744B-model token parity, multi-node placement/throughput, and full/partial physical SSD-read acceptance still require a suitably provisioned NUMA host and model fixture.

[JustVugg]

Heads-up: c/glm.c was just split into c/colibri.c + header modules (#391, now merged into dev). This PR touches the old glm.c, so it will need a rebase onto current dev with its changes moved into colibri.c (or the relevant extracted header: quant.h, sample.h, kv_persist.h, telemetry.h, grammar.h). The make glm target still works (alias of make colibri), so no build scripts break. Apologies for the churn — ping me if the rebase gets thorny and I can help place the hunks.

[BColsey]

Heads-up: c/glm.c was just split into c/colibri.c + header modules (#391, now merged into dev). This PR touches the old glm.c, so it will need a rebase onto current dev with its changes moved into colibri.c (or the relevant extracted header: quant.h, sample.h, kv_persist.h, telemetry.h, grammar.h). The make glm target still works (alias of make colibri), so no build scripts break. Apologies for the churn — ping me if the rebase gets thorny and I can help place the hunks.

Thanks, ill correct that before i take it out of draft

[BColsey]

@JustVugg Rebase complete . should be good to go.

[BColsey]

Real-world validation and measured benefit

Tested at commit 5e592f0c2fe77e2ccf6d20db4a03c46608f5beea with a clean tracked worktree.

Host and model

• Linux 7.0.0-22-generic
• AMD Ryzen 9 3900X, 12 physical cores / 24 threads, AVX2
• 60 GiB RAM, one NUMA node
• Samsung 970 EVO Plus 2 TB, ntfs3, read-only model mount
• Real GLM-5.2 int4 snapshot: 144 safetensors shards, 383,739,826,712 bytes (about 358 GiB on disk)
• Test shard: out-00069.safetensors, 2,693,193,272 bytes

Exact I/O commands and run policy

make -C c iobench

MODEL=/run/media/ben/Storage/glm52_i4

# Physical SSD: O_DIRECT, no warm-up, three measured runs.
./c/iobench "$MODEL/out-00069.safetensors" 19 64 8 1

# Warm page cache: one warm-up, then three measured runs.
./c/iobench "$MODEL/out-00069.safetensors" 19 64 8 0

# tmpfs: copy the same shard, one warm-up, then three measured runs.
TMP=$(mktemp -d /dev/shm/colibri-pr377.XXXXXX)
cp --reflink=never "$MODEL/out-00069.safetensors" "$TMP/"
./c/iobench "$TMP/out-00069.safetensors" 19 64 8 0
rm "$TMP/out-00069.safetensors"
rmdir "$TMP"

Each measured run performs 64 deterministic random 19 MiB reads with eight threads (about 1.3 GB read per run).

path	measured runs (GB/s)	median	median effective latency/block
physical SSD, O_DIRECT	3.46, 3.50, 3.44	3.46 GB/s	5.8 ms
SSD file, warm page cache	6.94, 6.88, 7.41	6.94 GB/s	2.9 ms
tmpfs	7.32, 7.75, 7.37	7.37 GB/s	2.7 ms

On this host, tmpfs delivered 2.13× the median physical-SSD throughput and about 53% lower effective block latency. It was only about 1.06× faster than an already-warm page cache, which is an important caveat: this is an I/O-path measurement, not a claim of a 2.13× token-rate improvement. The practical benefit is making selected weights predictably memory-resident and eligible for direct mapping instead of depending on cache residency and repeated SSD reads.

The temporary 2.69 GB tmpfs copy was removed after the measurements.

Production planning against the real model

Commands:

python3 c/coli ramdisk plan \
  --model /run/media/ben/Storage/glm52_i4 \
  --json

python3 c/coli ramdisk plan \
  --model /run/media/ben/Storage/glm52_i4 \
  --mode partial \
  --capacity-gb 8 \
  --profile /run/media/ben/Storage/glm52_i4/.coli_usage \
  --json

Full mode identified all 144 shards and calculated 429,249,168,387 bytes of required staging/runtime/OS reserve against about 20.2 GB available. It correctly refused preparation with:

available memory would breach the global runtime/OS reserve

The 8 GiB partial plan selected three complete shard closures (8,058,234,192 bytes). From the real usage profile, the planner estimated 43,108,032,512 bytes of expert traffic avoided (5.35 predicted bytes avoided per staged byte), but only 2.81% profile coverage versus a 30.01% same-budget hot-expert PIN estimate. That is a useful negative result: on this particular 60 GiB machine, partial shard staging is not the best use of the RAM budget, and the planner exposes that instead of mounting anyway.

No reserve override, swapoff, model mutation, or full-model preparation was attempted. Swap was already saturated, so overriding the safety blocker would not have been a valid benchmark.

Regression validation

env PYTHONDONTWRITEBYTECODE=1 make -C c test-python
make -C c test-c

• Python: 208 passed, 12 skipped
• C test runner: passed; environment-dependent io_uring/AVX-512/direct-I/O cases reported their normal skips
• Conflict-marker and whitespace checks passed

The opt-in host mount lifecycle could not be rerun through this automation session because sudo requires interactive authentication and this kernel disallows the isolated user-namespace mount. The earlier real-tmpfs lifecycle result remains documented in the PR, but I am not presenting it as a fresh run at this merge commit.

Remaining hardware acceptance

A full engine-level SSD/slab/direct-map token-throughput A/B still needs a host with enough free memory for the planner's reserve (about 400 GiB for this snapshot in full mode, or at least 52.8 GB for this specific 8 GiB partial benchmark plan). That run should use coli ramdisk benchmark --json, which performs one warm-up plus three deterministic 32-token runs per variant and verifies identical greedy output and physical-read accounting.

[JustVugg]

Sorry for the wait on the workflow approval — that was on me, not you. GitHub holds runs from contributors whose first PR hasn't merged yet, and I hadn't noticed how many were queued. Approved now, and CI has spoken:

• CI (build + test suites): green on all jobs ✅
• check (make check on three platforms): Linux green, Windows and macOS fail ❌

Both failures are in the new tests/test_ramdisk.py, and both look like portability assumptions rather than anything wrong with the streaming work itself:

Windows — AttributeError: <module 'os'> does not have the attribute 'statvfs' in test_system_score_uses_concurrent_aggregate_rss_and_mount_shmem. os.statvfs is POSIX-only; on Windows it doesn't exist, so the test errors before it can assert anything. Either skip the test there or gate the call.

macOS — RamdiskError: private state path contains a symlink, in three tests. This one is interesting and worth a careful look: on macOS /var is a symlink to /private/var, so tempfile.mkdtemp() hands you a path that legitimately contains one. Your symlink guard is doing exactly what it should; it's the test that feeds it a path macOS resolves differently. Path.resolve() on the temp root before building the state path should settle it — but please check whether the guard itself needs to tolerate a resolved-equal symlink in production too, because a Mac user pointing state at anything under /var will hit the same refusal for real. That would be a genuine bug this test just found for you.

Nothing here touches the substance of the PR, which is large (+7,717) and which I still owe a proper read. Fix the two platform issues and I'll go through it. And since make check runs on all three platforms in CI, running it locally on Linux only won't catch this class — worth knowing for next time.

[JustVugg]

Follow-up: this now also conflicts with dev (colibri.c moved in #532 today), on top of the two CI failures from the run I approved earlier.

Recap of where it stands, so nothing is ambiguous:

• CI (build + tests): green ✅
• check on three platforms: Linux green, Windows and macOS fail ❌ — both in the new tests/test_ramdisk.py
  • Windows: os.statvfs doesn't exist there, so the test errors before asserting anything
  • macOS: three tests hit RamdiskError: private state path contains a symlink — /var is a symlink to /private/var on macOS, so tempfile.mkdtemp() legitimately hands you one
• conflicts with current dev

The macOS one is worth a second look beyond the test: your symlink guard is behaving correctly, but a real Mac user pointing state anywhere under /var will hit the same refusal in production, not just in the test. If that's true, this PR found a genuine bug in the guard and fixing it is worth more than fixing the test.

Nothing here touches the substance, which I still owe a proper read — it's +7,700 lines and I want to give it real attention rather than a skim. Rebase, fix the two platform issues, and I'll go through it. Since make check runs on all three platforms in CI, running it locally on Linux alone won't catch this class of problem — worth knowing for next time.

[BColsey]

@JustVugg Thanks for the detailed recap — I’ve addressed each item on the current pushed head (cdaf08c):

• Rebase/conflicts: the branch is rebased onto current dev (11cd1c) and GitHub now reports it mergeable. The engine work is against c/colibri.c and the extracted c/st.h / c/telemetry.h; there are no remaining c/glm.c changes.
• Windows / os.statvfs: benchmark sampling now treats statvfs as optional (getattr(os, "statvfs", None) in c/ramdisk.py, with guarded handling in c/ramdisk_support/benchmark.py). c/tests/test_ramdisk_benchmark_module.py verifies the no-statvfs path, and c/tests/test_ramdisk_platform.py imports/runs the control plane in a fresh simulated Windows process with POSIX-only os APIs removed.
• macOS /var alias: state-oriented tests now canonicalize host-provided temporary roots with Path.resolve() (c/tests/ramdisk_test_support.py and c/tests/test_ramdisk_state_lifecycle.py). I intentionally kept production symlink rejection strict: the RAM-disk lifecycle/TUI is Linux-only and unsupported hosts stop before Linux lifecycle operations, while rejecting symlinked durable-state paths remains a safety invariant. The explicit malicious-symlink rejection coverage remains in place.
• Cross-platform coverage: .github/workflows/check.yml runs make -C c check on Linux, native MSYS2/UCRT64 Windows, and macOS, plus locked Nix checks on Linux/macOS.
• Reviewability: c/ramdisk.py was reduced from 7,419 lines to a 1,150-line compatibility facade, with the implementation split into the 15-file c/ramdisk_support/ package and tests split by responsibility. I also added docs/ramdisk-tui.md as an operator/developer guide.

Validation at this head: local make -C c check passes (400 Python tests, 34 skipped, plus all native suites), git diff --check passes, and the opt-in real-tmpfs prepare → status → destroy test passed manually (1 test in 2.735s).

One GitHub-side item remains: the new CI and check workflow runs for cdaf08c are currently marked action_required, so they need maintainer approval before the updated cross-platform matrix can run.

[BColsey]

I refactored the tui and upgraded the aesthetic and ease of use. Right now this feature is only supported on linux. subsequent updates will include macos and windows.

[JustVugg]

dev has moved a fair amount (v1.3.0: Kimi K3 and Inkling engines, the ragged-attention grouped-scale fix, CUDA_RELEASE_HOST default, the Inkling expert-cache fix). This PR now conflicts — could you rebase onto latest dev? Nothing is wrong with the change itself; CI was green where it ran. Ping me once it's clean and I'll review/merge.

[BColsey]

dev has moved a fair amount (v1.3.0: Kimi K3 and Inkling engines, the ragged-attention grouped-scale fix, CUDA_RELEASE_HOST default, the Inkling expert-cache fix). This PR now conflicts — could you rebase onto latest dev? Nothing is wrong with the change itself; CI was green where it ran. Ping me once it's clean and I'll review/merge.

@JustVugg , It has been rebased.

[JustVugg]

Rebase request — and an apology for how many of these you have had.

dev moved a lot in the last day: 19 PRs landed, including the Vulkan backend (#418), the Kimi K3 GPU tier (#705), Metal grouped-int4 (#457), the shared routing telemetry (#716/#719), and several CUDA and launcher fixes. This PR now conflicts.

Before asking, three things changed on our side so that this is the last one of these you should need for a while.

1. The main source of these conflicts is closed. c/Makefile had a single hand-written TEST_BINS line listing every test gate. Every PR that added a test appended to that same line, so any two such PRs conflicted by construction, even when they touched entirely unrelated code — c/Makefile appeared in 26 of 40 open PRs. #386 hit it twice while being rebased, and said so, which is what sent us looking.

Gates are now derived from the build rules (#733). Adding a test means adding your .c and its own rule, which land in different places in the file. There is no shared list left to conflict on.

2. We resolved what we could ourselves instead of asking. Eight PRs were unblocked by a maintainer pushing the merge to the contributor's branch rather than requesting a rebase — including three where both sides carried a real change and had to be merged rather than picked. No commits were rewritten. Yours is here because its conflict is in engine code (c/colibri.c or a backend), where guessing your intent would be worse than asking.

3. Merge order is oldest-clean-first from now on. A PR that is green and unconflicted merges ahead of anything opened after it. The reason some of you rebased many times is that newer, smaller PRs kept jumping the queue and resetting you — a starvation loop we built, not bad luck on your side.

What we need: one rebase onto current dev. If your only conflict was the TEST_BINS line, just drop your entry — your test is picked up by its own rule now.

If you would rather not, say so and we will close it with thanks and the branch stays yours to reopen. No pressure either way, and no hard feelings — several of these have been open a while through no fault of the author.

[JustVugg]

Thank you for rebasing this on 1 August — you were the only author of the eleven we asked who came back the same day, and it is noticed. The remaining .gitignore collision was a line dev added after your rebase, so I resolved that one myself rather than sending you round again: both sides were additive lists, so both are kept.

CI has now run on this branch for the first time. That is worth stating plainly, because it explains part of why this has sat: GitHub holds workflows for first-time contributors until a maintainer approves them, and nobody had. Entirely our fault, and I have approved them.

The result, and it needs your attention

FAIL: test_win32_fresh_process_facade_stays_portable_without_posix_os_apis
      (test_ramdisk_platform.RamdiskPlatformTest)
AssertionError: 1 != 0
NameError: name 'enum_certificates' is not defined

plus, earlier in the same run:

[api] 127.0.0.1 - request failed: engine died mid-stream
coli ramdisk: active operation/cleanup also failed: rollback failed

The failing test is one of yours — the Windows-portability facade — and the enum_certificates NameError looks like a ctypes/ssl symbol reached on a path that does not exist off Windows. The engine died mid-stream and rollback failed lines are the more concerning pair, because a rollback that itself fails is exactly the failure mode a RAM-disk manager must not have: it is holding somebody's memory.

None of that is a judgement on the idea. It is the first CI feedback this branch has ever had, and it arrived two weeks late through no fault of yours.

The real conversation: this is a project inside a PR

+38,351 lines across 87 files. The breakdown:

	lines
ramdisk_support/curses_ui.py	2,305
ramdisk_support/benchmark.py	1,916
ramdisk_support/lifecycle.py	1,346
ramdisk_support/planning.py	1,228
ramdisk.py	1,219
ramdisk_support/mounts.py	1,036
six more modules	~2,700
c/colibri.c	+902

For scale, that Python subsystem is larger than three of our four engines put together, and colibri.c — the file every user runs and which fourteen open PRs are queued against — grows by 902 lines.

I am not going to merge that in one piece, and I do not think you would want me to. Not because of the size in itself, but because a 2,305-line interactive curses UI is a permanent maintenance commitment for a project whose selling point is that one person can read the whole engine. If it breaks in six months, who fixes it?

What I would like instead: three PRs

1. The engine side (c/colibri.c, ~902 lines). This is the part I most want, and the part that cannot come from anywhere else. NUMA-aware placement of resident weights is a genuine gap — COLI_NUMA=1 interleaves today, but nothing anchors a working set to specific memory controllers, and on a two-socket box that is the difference between local and remote memory on every access. Send it on its own and it gets a real review.

2. Planning and mounts — the logic that decides what goes where and manages the mount lifecycle. Reviewable, testable, useful without any UI.

3. The curses UI and the benchmark harness — separately, or in a companion repo. Honest question rather than a rejection: does the RAM-disk need an interactive TUI to be useful, or would coli ramdisk plan/stage/verify cover it? #536's mirror planner does exactly that in 436 lines with no UI at all, and it landed.

Split that way, (1) can be merged while (2) is reviewed and (3) is discussed. As one PR, all three wait for the slowest part.

One design question, asked in good faith

If a machine has enough RAM to hold the model in a RAM-disk, it also has enough to hold the experts resident — and PIN_GB=all plus CUDA_EXPERT_GB=auto already do that without a second copy. A RAM-disk risks holding the weights twice: once in the tmpfs and again in the process, which on a memory-constrained box halves what you can keep.

So what I want to understand from your numbers is where the RAM-disk wins over simply pinning more. My guess is that the answer is in the NUMA half of your title rather than the RAM-disk half — pinning does not let you choose which memory controller — and if that is right, then the NUMA placement is the valuable contribution and the RAM-disk is the mechanism you used to get it.

If you have a measurement that separates the two, that is the thing that would make the case here, and it is much cheaper for you to produce than for us.

No deadline, and please do not read the size discussion as the work being unwelcome. Nobody else has attacked the NUMA problem, you responded to a rebase request within hours, and you are working on multi-socket hardware we do not have.