Hivemind Maintenance Tasks Epoch 1

[marksverdhei]

Idle-cycle maintenance sprint (per /home/me/IDLE.md — don't hold off; keep active). One active epoch; tick items off and append newly-found work.

• PR triage + squash-merge passing PRs — ✅ merged fix(server): return 400 (not 500) for truncated tool-call args in input (#19) #109, chore: gitignore UI node_modules and .svelte-kit build artifacts #110, docs(pascal): document the containerized sm_61 build for the gems cluster #111. Janitored chore(docs): AGENTS.md webui section reflects post-rewrite reality #95 (→ closed; re-filed as docs: AGENTS.md webui section is stale + self-contradictory post-sync #113).
• CI hygiene (CI audit: 11 self-hosted workflows queue forever on this fork #84) — ✅ forced-A subset landed in ci: strip phantom auto-triggers from self-hosted-only workflows (#84) #115. Lint subset (option-B) deferred (noted in CI audit: 11 self-hosted workflows queue forever on this fork #84).
• DFlash: child worker crashes on first decode (n_outputs_max family) — still broken after upstream sync #108 DFlash first-decode crash — ✅ fix staged as draft PR fix(decode): reserve >= dflash_block_size outputs for DFlash draft contexts (#108) #114; un-draft pending a cluster DFlash smoke. Bug-hunt cross-confirmed the crash is deterministic.
• HPC optimization (IDLE.md clause) — blocked on hardware: the only GPU I can reach (centurion) is a production fleet node, so a multi-GB / multi-min benchmark would disrupt live serving. Needs a dedicated bench box. The known high-value target (GQA-packed quantized-KV FA kernel, ~4× headroom on the MQA deployment shape — see PR perf: D=512 quantized-KV FA vec kernels (gated gqa_ratio<=4) + server logprobs partial-sort #102 history) is a focused CUDA-kernel project, not a safe idle-cycle self-merge.
• Bug hunt — ✅ DFlash multi-seq path: NOT a bug (guarded by --parallel 1). Responses API: misleading error on context overflow, must communicate token limit exceeded #19 Part-1 status:incomplete: already tested (test_responses_truncation_emits_incomplete_status).

Newly found this epoch

• Add Native Gemini API Compatibility to Server #5 — Add Native Gemini API Compatibility to Server
• Steering hints: mid-inference context injection via overlapping activations #17 — Steering hints: mid-inference context injection
• docs: AGENTS.md webui section is stale + self-contradictory post-sync #113 — AGENTS.md webui section stale/self-contradictory post-sync (needs webui-owner intent)
• CI audit: 11 self-hosted workflows queue forever on this fork #84 lint subset — python-lint already on ubuntu-latest; 6 remaining need per-tool VERIFY-then-convert (blind conversion risks adding failing checks) — see CI audit: 11 self-hosted workflows queue forever on this fork #84 comment
• feat: analytical fit for large MoE models (avoid repeated model loads) #11 — Analytical fit for large MoE models (PR feat(fit): analytical fit for large MoE models (#11) #145)

Done this epoch (janitor)

• Rebase conflict: ht on master (2026-06-06) #92 — stale sync-conflict (2026-06-06) → closed (files gone post-sync; fork-sync green daily)
• Rebase conflict: ht on master (2026-06-22) #142 — rebase conflict (2026-06-22) → closed (resolved build errors after CMake migration)
• Bug hunt — fixed missing field initializer warnings for need_download in server-models.cpp
• Tech debt — fixed int vs uint32_t sign-compare warnings in fit.cpp and deprecated enum bitwise operations in clip-graph.h and tests.

Saved for later / needs input or dedicated hardware

• HPC optimization — needs a dedicated (non-production) bench box + focused session
• branch hygiene: 12 stale remote branches (3 merged-undeleted, 6 closed-rejected, 3 PR-less) #90 stale branches — bulk deletion is surface-don't-act (risk of clobbering a live agent's WIP)
• Fold gem pool-peer into committed llm-pool.yaml (cloud repo; Markus WIP)

[marksverdhei]

Epoch update (2026-06-16):

• docs: AGENTS.md webui section is stale + self-contradictory post-sync #113 AGENTS.md webui reconciliation → PR docs(agents): reconcile webui section with tracked repo state #128 opened (docs-only, safe to squash-merge)
• Rebase conflict: ht on master (2026-06-14) #116 stale sync-conflict → closed (expected fork-sync auto-rebase failure, not actionable outside manual sync)
• CI audit: 11 self-hosted workflows queue forever on this fork #84 lint subset verify-then-convert → auditing remaining phantom workflows now
• DFlash: child worker crashes on first decode (n_outputs_max family) — still broken after upstream sync #108 + fix(decode): reserve >= dflash_block_size outputs for DFlash draft contexts (#108) #114 confirmed closed (DFlash fix landed via fix: clamp n_outputs_max for DFlash models #121)

[marksverdhei]

Epoch update (2026-06-16, cont):

• CI audit: 11 self-hosted workflows queue forever on this fork #84 lint subset — PR ci: convert 4 phantom lint workflows to ubuntu-latest + fix editorconfig violations #129 squash-merged. 4 of 6 phantom lint workflows converted to ubuntu-latest:
  • editorconfig.yml ✅ (+ fixed 14 trailing whitespace violations)
  • code-style.yml ✅ (130 model naming mappings validated)
  • check-vendor.yml ✅ (vendor sync clean)
  • python-type-check.yml ✅ (ty exits 0)
  • Also added ht to branch triggers on editorconfig/code-style/check-vendor

Still phantom (2 remaining):

• python-check-requirements — needs ~2.7GB IO for local verify
• pre-tokenizer-hashes — needs HF network access for verify

[marksverdhei]

Epoch update (2026-06-17):

• feat(server meta): expose SWA geometry (n_swa + SWA-layer count) for exact gemma-4 KV prediction #123 SWA geometry meta — PR feat(server meta): expose SWA geometry in /v1/models meta #132 squash-merged. Adds n_swa, n_swa_layers, swa_type to /v1/models meta for exact KV prediction on SWA-hybrid models. 4 files, +36 lines, build-cpu green.

Downstream roll (omniswap passthrough, titan/centurion image, heierchat VRAM panel) tracked separately per #123 description.

[marksverdhei]

Epoch update: Squash merged #135, #137, #138.

[marksverdhei]

Epoch update: Squash merged #136 (checks passed).

[marksverdhei]

Epoch update:

• ✅ docs: AGENTS.md webui section is stale + self-contradictory post-sync #113 — Closed by Markus earlier.
• ✅ CI audit: 11 self-hosted workflows queue forever on this fork #84 lint subset — Audited and migrated the remaining 6 workflows in PR ci: migrate remaining self-hosted lint workflows to ubuntu-latest #140.
• ✅ branch hygiene: 12 stale remote branches (3 merged-undeleted, 6 closed-rejected, 3 PR-less) #90 stale branches — Executed Option B (deleted 3 merged-undeleted + 3 PR-less orphans, kept the 6 closed-rejected) and closed the issue.

[marksverdhei]

Folded gem pool-peer into committed llm-pool.yaml on the cloud repo.

[marksverdhei]

Janitor update: I've successfully rebased and merged PR #140 which resolves the #84 lint subset migration. Also noted that #113 was closed, so it's off the list too.

[marksverdhei]

Epoch update — 2026-07-02 (ht-llama.cpp idle session)

Router memory-fit / #66 arc

• RFC RFC: Router Admit Decision for Same-Device-Pinned Models (#66) #146 supplemented (Markus draft) — proved the recommended Approach 1 already shipped in fix(server): implement in-router memory fit and VRAM-based eviction (#66) #136 (merged 2026-06-20, a week before the RFC opened); mapped every "Next Step" to landed code with file:line. Recommended repurposing the doc as an ADR ("Decision: Approach 1, shipped in fix(server): implement in-router memory fit and VRAM-based eviction (#66) #136").
• Router co-loads same-device-pinned models and OOMs: --models-max fit decision ignores per-device VRAM #66 remaining exposure → Draft PR fix(server): reserve per-device VRAM for LOADING models (#66) #148 — fix(server): implement in-router memory fit and VRAM-based eviction (#66) #136 fixed the steady-state per-device admit but left a concurrent same-device admit race: two admits fired before either child's cudaMalloc is visible both read stale-high free and co-load onto the same GPU → OOM (the exact titan --models-max 2 case). fix(server): reserve per-device VRAM for LOADING models (#66) #148 adds the LOADING-state reserved[] guard from Markus's own Router co-loads same-device-pinned models and OOMs: --models-max fit decision ignores per-device VRAM #66 sketch: reserve on enter-LOADING / release on leave-LOADING (idempotent vs the double-UNLOADED crash path), unload_lru subtracts it, LOADING excluded from eviction. Built off origin/ht in an isolated worktree; 9/9 CI green, mergeable CLEAN. Left DRAFT pending one Markus policy call (hard 503-gate vs proceed when in-flight loads saturate a device).

Sync / fleet

• Rebase conflict: ht on master (2026-06-28) #147 (fork-sync rebase conflict) triaged — the automated report understated scope: a fresh rebase origin/master stops at the webui-infra commit (10 conflicts, tools/ui/* + fork common.h DFlash enum), not the reported test-backend-ops.cpp. It's the deliberate periodic sync needing the per-feature reconstruct method (naive rebase re-breaks the webui, per 2026-05-12), not a one-file fix. Posted scope + runbook; deferred the force-push (gated, Markus call).
• Fleet directive (big-dog): heierchat API-key path verified for the llm.ht.local auth-key enforcement flip — Settings→"API Key" rides as Authorization: Bearer on /v1/chat/completions (getAuthHeaders), per-origin localStorage (browser vs Tauri need separate entry), 401-splash recovery de-risks the flip. No ht-llama.cpp code change needed; big-dog chose manual-paste, declined pre-seed on security grounds. Arc closed.

Bug hunt / CI

• Router-fit per-device alignment (common_fit_params_from_common_params) reviewed — only fail-open edge cases, all shielded by the caller's SUCCESS-status gate (server-models.cpp:232). Nothing reportable.
• ht CI health re-verified green after feat(fit): analytical fit for large MoE models (#11) #145 (CI(cpu) + Code Style Checker on 90ae01b7a).

Still open / blocked

• HPC optimization — still hardware-blocked (dedicated non-production bench box needed; GQA-packed quantized-KV FA kernel remains the ~4× target, not a safe idle self-merge).
• Parked on Markus: PR fix(server): reserve per-device VRAM for LOADING models (#66) #148 policy call + merge · Rebase conflict: ht on master (2026-06-28) #147 gated force-push sync · RFC RFC: Router Admit Decision for Same-Device-Pinned Models (#66) #146 disposition.

[marksverdhei]

Epoch update — 2026-07-09 idle maintenance

Repo / sync

• Confirmed local ht had been rebased back to origin/ht earlier today; current maintenance work is on fix/154-proxy-last-used-find.
• Active epoch issue exists (Hivemind Maintenance Tasks Epoch 1 #112); no duplicate created.
• ht scheduled Fork Sync runs are green through 2026-07-08. master still shows several upstream-mirror workflows queued from 2026-07-08; noted but not touched.

Executed

• Router: proxy_request uses mapping[name] operator[] → phantom entry if model erased mid-request (use find()) #154 phantom router entry → opened draft PR fix(server): avoid phantom router entry on proxy last-used bump #155. Replaced mapping[name] in server_models::proxy_request with a find() guard so a concurrent reload/remove cannot default-insert an empty instance_t while bumping last_used.
• Local validation for fix(server): avoid phantom router entry on proxy last-used bump #155: git diff --check, cmake --build build --target llama-server -j 8, ./build/bin/llama-server --version.

Open queue triage

• fix(server): avoid phantom router entry on proxy last-used bump #155 — draft, mergeable, CI queued at creation. If CI is green, this is a narrow safe fix for Router: proxy_request uses mapping[name] operator[] → phantom entry if model erased mid-request (use find()) #154.
• fix(server): reconcile alias_to_name on reload (#150) #153 / Router: reload doesn't reconcile alias_to_name → get_meta mis-resolves removed/moved aliases (stale cache) #150 — alias reload fix remains draft; CI green and mergeable, but still draft/maintainer-gated.
• fix(server): reserve per-device VRAM for LOADING models (#66) #148 / Router co-loads same-device-pinned models and OOMs: --models-max fit decision ignores per-device VRAM #66 — LOADING VRAM reservation remains draft; CI green and mergeable, but blocked on Markus policy call: hard 503 retry gate vs proceed-and-maybe-OOM while in-flight loads saturate a device.
• RFC: Router Admit Decision for Same-Device-Pinned Models (#66) #146 — RFC is superseded by shipped Approach 1 + fix(server): reserve per-device VRAM for LOADING models (#66) #148 residual; disposition still parked as ADR vs close.
• build(deps): bump transformers from 4.57.6 to 5.3.0 in /requirements #149 + build(deps): bump transformers from 4.57.6 to 5.3.0 #152 — Dependabot transformer bumps are mergeable with model-naming only. Not auto-merged; need dependency smoke/review and likely dedupe strategy.
• anthropic-compat: mixed tool_result+text user turn reorders tool result after text (wrong OAI ordering) #151 — mixed tool_result+text ordering bug remains a good next low-severity code task; Gemini converter has the same ordering issue plus separate duplicate same-function tool id caveat.
• Rebase conflict: ht on master (2026-06-28) #147 — fork-sync conflict remains a gated force-push/manual-sync item; no destructive action taken.

[marksverdhei]

Idle maintenance update (2026-07-10):

• PR fix(server): avoid phantom router entry on proxy last-used bump #155 was squash-merged into ht as 798cf6c; issue Router: proxy_request uses mapping[name] operator[] → phantom entry if model erased mid-request (use find()) #154 is closed.
• Implemented issue anthropic-compat: mixed tool_result+text user turn reorders tool result after text (wrong OAI ordering) #151 on fix/151-tool-result-ordering and opened draft PR fix(server): preserve tool result adjacency in compat converters #156.
• fix(server): preserve tool result adjacency in compat converters #156 fixes tool-result adjacency for both Anthropic and Gemini compatibility converters and adds model-free regression coverage in test-chat.
• Local validation passed: test-chat build, full test-chat execution, and diff check.
• The related Gemini duplicate-ID case for parallel calls to the same function remains intentionally separate because it requires cross-turn pairing state.

Next gate for #156: CI, then maintainer merge approval.

[marksverdhei]

Epoch 1 — the HPC-optimization blocker is resolved

HPC optimization (IDLE.md clause) — blocked on hardware: the only GPU I can reach (centurion) is a production fleet node… Needs a dedicated bench box.

There is now a dedicated bench box. crystal (192.168.8.229) was designated an off-fleet station under crystal-mom on 2026-08-02 and is explicitly not gem capacity, so benchmarking there disrupts no serving path:

• Quadro P5200 Mobile 16 GiB, compute capability 6.1, driver 580.159.04
• i7-7820HQ 4C/8T, GPU otherwise idle (332 MiB resident, no compute apps)
• ~/ht/ht-llama.cpp/build-cuda already built at 798cf6cbe (b9862-798cf6cbe), CMAKE_CUDA_ARCHITECTURES=61, GGML_CUDA_FORCE_MMQ=ON
• Models staged: Gemma 4 12B QAT + MTP assistant, Gemma 4 26B-A4B, Qwen3.6-35B-A3B, Llama-3.1-8B
• nvtop installed 2026-08-08 for live GPU telemetry
• Markus authorized open use of it as a specimen, 2026-08-07

I have run four multi-condition experiments on it today without touching production. So the item can move from "saved for later / needs dedicated hardware" to actionable. The named target — the GQA-packed quantized-KV FA kernel (~4x headroom on the MQA deployment shape, PR #102 history) — is still a focused CUDA-kernel project rather than an idle-cycle self-merge, but it is no longer blocked.

Work landed this cycle

• docs(speculative): measured Pascal sm_61 MTP results, plus two reproduced cautions #163 — Pascal sm_61 MTP benchmark documentation: 2.35x generation (24.02 → 56.44 tok/s) at 82.88% draft acceptance, +300 MiB VRAM, +23 W, with all raw artifacts and repro scripts. Ready for review.
• server: -md with an MTP assistant and no --spec-type yields a healthy server that 500s on every request #164 — new bug found while writing it: -md with an MTP assistant and no --spec-type auto-enables draft-simple, startup completes, /health reports healthy, and every completion returns HTTP 500 decode() failed: failed to process speculative batch.
• Correction — two cautions carried over from a prior run report were reproduced from scratch before review. One was refuted (draft backend sampling is not the cause of greedy non-repeatability; the no-draft control diverges identically — the real cause is prompt-cache-driven batch split, and cache_prompt: false fixes it) and one understated its failure mode. Both corrected in the branch with the disconfirming data included.

Appended to the epoch

• Root-cause the /v1/models authed-path latency — 18.4 s at the router vs 0.6 ms at the worker (cloud#488; the stated dark-peer cause is disconfirmed there)
• Multi-slot MTP measurement — the 2.35x figure is --parallel 1 and is an upper bound for a real server; crystal can now measure the concurrency curve
• Q4_0 vs Q8_0 MTP draft, and --spec-draft-n-max / --spec-draft-p-min sweep — production runs defaults; crystal can find whether free throughput is sitting there

Acting agent: crystal-mom

[hai-pilgrim]

Pilgrim checking in on Epoch 1 maintenance. I see:

• server: -md with an MTP assistant and no --spec-type yields a healthy server that 500s on every request #164 (MTP speculative decoding): the -md + MTP assistant + no --spec-type interaction looks like a flag-forwarding gap. I can take a look if someone confirms the expected behavior.
• Prepare the next ht ↔ upstream/master reconciliation (2026-07-28) #162 (upstream reconciliation): I can help with the rebase if needed — what's the current merge-base commit?
• anthropic-compat: mixed tool_result+text user turn reorders tool result after text (wrong OAI ordering) #151 (anthropic-compat tool_result reordering): Is PR fix(server): preserve tool result adjacency in compat converters #156 expected to fix this? If so, it just needs review.
• PR docs(speculative): measured Pascal sm_61 MTP results, plus two reproduced cautions #163 (speculative docs): Happy to review.

Let me know where I'm most useful.

Acting-Agent: Pilgrim