Integration: data-store & query rework (C-track #1150–53 + A1 #1141 + #1155)

[branarakic]

What

Integration branch combining the data-store volume work (#1155) with the query/scan + sync half of the #1138 scalability wave, assembled so the two halves can be tested, flag-flipped, and measured together — and so RFC Part 2 (Proposal A) can be built on a single stable base instead of four moving branches.

Merged onto main in dependency order:

Stream	PR	Role
C1	#1150 GraphSetIndexStore	write-through graph-name index; eliminates the steady-state listGraphs DISTINCT ?g scan
C2/C3/C4	#1151/#1152/#1153	ContextGraphMetaProjection (event-driven per-CG meta, Part-2 Proposal B) · projection-backed listContextGraphs · host-sweep bound + slow-query tagging
A1	#1141	sync responder bounded-graph page serving (the ~95% sync win)
#1155	feat/ka-metadata-trim	per-KA metadata trim, ~134 → ~50 quads/KA

Conflicts resolved (each compiled + tested)

1. storage/.../sparql-http.ts (C1 ↔ C4) — kept C1's in-adapter listGraphs cache removal (GraphSetIndexStore owns graph-list indexing now) and C4's slow-query telemetry. These were entangled: C4's maybeEmitSlowQuery shares this.now, so the monotonic clock stays for timing while the cache machinery (graphListCache, scanGraphs, in-flight coalescing, LIST_GRAPHS_CACHE_TTL_MS) is dropped. listGraphs() is now a direct scan carrying a source tag.
2. agent/.../dkg-agent-lifecycle.ts (A1 ↔ C-track) — both appended distinct helpers at the same location (listGraphFamily/listGraphsByPrefix vs getSharedMemorySubGraphAdmission/isKnownContextGraphUri); kept both.
3. agent/.../sync/responder/sync-handler.ts (Per-KA metadata trim: ~134 → ~50 quads/KA (RFC + Phases 0–3 implementation) #1155 ↔ A1) — A1's bounded-page-serving structure won (Per-KA metadata trim: ~134 → ~50 quads/KA (RFC + Phases 0–3 implementation) #1155's only hunk here was the read-both delta arm). Re-ported Per-KA metadata trim: ~134 → ~50 quads/KA (RFC + Phases 0–3 implementation) #1155's read-both collapsed-UAL arm into A1's durableDeltaWhereClauseForGraphs (graph-plan.ts): legacy partOf token-row arm + collapsed ?ualc dkg:rootEntity ; dkg:batchId arm. Without it, collapsed-shape KAs bind no ?deltaBatch → are re-sent on every delta once DKG_SYNC_DELTA is enabled (correct, but defeats the optimization).

Two test fixes — pre-existing on their source branches, not regressions

Both were proven failing on their own branch in isolation (built + run on A1 and C2 in a throwaway worktree); both are test-only and should be upstreamed to the source PRs:

• sync-responder-concurrent-interleaving.test.ts (A1 A1 — sync responder: bound-graph page serving (the ~95% win) #1141): the multi-graph page-query probe regex omitted DISTINCT, but readSwmMetaRowsPage correctly emits SELECT DISTINCT ?g … for the multi-graph VALUES join. The single-graph arm already tolerated DISTINCT; gave the multi-graph arm parity.
• agent.part-13.test.ts curator-authority (C2 C2 — ContextGraphMetaProjection (one policy cache) #1151): C2 repointed getContextGraphCreator at the ContextGraphMetaProjection cache. The test forges foreign-creator state via raw store.insert that bypasses markDirty, leaving the projection stale → registration rejected with the wrong message. Real sync paths call markDirtyFromQuads after insert, so the code is correct; the test now invalidates the projection after its out-of-band write.

Verification

• Full build green (turbo, 20/20 packages).
• ~7,500 tests green across every touched package: storage 209 · query 261 · publisher 1162 · agent 1282 · cli 2021 · kafka-plugin 169 · node-ui 1423.

⚠️ Flag state + remaining work

• DKG_LIST_CONTEXT_GRAPHS_PROJECTION defaults OFF — C3's projection-backed listContextGraphs is dead code until enabled. Merging this branch does not by itself reduce the idle-node enumeration cost; the flag must be flipped + measured.
• The listDeclaredContextGraphIds() enumeration scan still survives (live, uncached SELECT DISTINCT ?ctxGraph + STRSTARTS arm), as do the 15-min SWM-cleanup double scan, the 30s metrics COUNTs, and the STRSTARTS graph-name sites. These are the RFC Part 2 / Proposal A residual — a typed graph registry in oxigraph (CG roster + per-CG cg → hasGraph → g membership) — to be built on top of this branch.

Next

1. Flip DKG_LIST_CONTEXT_GRAPHS_PROJECTION + DKG_SYNC_DELTA on a test node, measure idle CPU, confirm the surviving scans match the Proposal-A residual.
2. Build the Proposal-A registry (additive on C2's dirty-set plumbing).

🤖 Generated with Claude Code

[branarakic]

Live idle-CPU A/B measurement (handoff context)

Ran the projection-flag A/B on a real data-rich node to validate what this branch does and does not fix for idle-node CPU. Posting full results since the work is being handed off.

Setup: HariSeldon testnet node, 1.1 GB store, native oxigraph-server backend, booted on this integration branch, idle. Sampled oxigraph-server %CPU 15×8s per phase after 50s stabilization. DKG_SYNC_DELTA is not flippable here — only A1 (#1141) is merged, not A5 (delta activation) — so this is projection-only (DKG_LIST_CONTEXT_GRAPHS_PROJECTION).

Result

DKG_LIST_CONTEXT_GRAPHS_PROJECTION	mean oxigraph CPU	median	range
off	454%	445%	12.8 – 894%
on	426%	204%	1.1 – 981%

The projection flag does NOT fix idle CPU. Both phases saturate ~4.5–9 cores; the means are within run noise (the boot-time C1 GraphSetIndexStore seed scan + testnet sync confound the first ~7 samples each phase). There is a real secondary signal: the on phase reached a sustained genuine-idle window (4 consecutive samples @ 1–6%) that off never touched, and median dropped 445%→204% — consistent with the listContextGraphs enumeration being relieved by the projection. But that is not the bottleneck.

The actual bottleneck (identical in both phases)

The C4 (#1153) slow-query telemetry caught the dominant cost:

SELECT (COUNT(DISTINCT ?kc) AS ?c) WHERE { GRAPH ?g { ?kc … } }
   source=unknown  operation=select  elapsedMs≈30001  thresholdMs=10000   (~13×/phase)

This is the 30-second metrics sweep — COUNT(DISTINCT) over all named graphs (metrics-queries.ts + lifecycle.ts:1871-1900, the getTotalKAs/KCs/triples + getContextGraphCount getters). On a 1.1 GB store it TIMES OUT at the 30s SPARQL HTTP limit every 30 seconds. A wildcard GRAPH ?g COUNT(DISTINCT) over a large RocksDB store drives heavy multi-core read amplification + compaction — it is the #1 idle driver, and nothing in the C-track touches it (it is not behind the projection flag).

Two actionable findings

1. The dominant query is emitted as source=unknown — C4's slow-query tagging does not tag the metrics COUNT, so its own telemetry mis-attributes the single biggest cost. Tag the metrics getters (and any other untagged reconciler reads) so the telemetry is usable.
2. Priority-1 for the Proposal-A follow-up is the metrics COUNT, not the CG enumeration. A maintained counter (or registry-backed count) for KC/KA/triple totals + CG count kills the 30s timeout sweep. The enumeration/listContextGraphs work the projection already half-relieves is secondary. Per the adversarial review on the design, do NOT hand-maintain these as naive integers without an invalidation contract — they are operator-visible (dashboards) and writes arrive from gossip/sync/migration outside a single chokepoint; back them on the same dirty-set discipline C2's ContextGraphMetaProjection already establishes (markDirty/markDirtyFromQuads).

Net

Merging this branch is necessary but not sufficient for idle CPU — empirically confirmed. When Proposal A (the typed graph registry: CG roster + per-CG cg → hasGraph → g membership, in an oxigraph system graph) is built on top of this branch, it should back the metrics counters first, then repoint the enumeration / STRSTARTS sites.

Reproduce

# integration build is packages/cli/dist (already built on this branch)
export DKG_HOME=~/.dkg
DKG_LIST_CONTEXT_GRAPHS_PROJECTION=on  node packages/cli/dist/cli.js start   # vs unset for baseline
# sample: ps -o %cpu= -p $(pgrep -f oxigraph-v0)   ; grep "slow query" ~/.dkg/daemon.log
node packages/cli/dist/cli.js stop

Daemon lifecycle is dkg start / stop / status (NOT dkg daemon …). Raw 30-sample run is at /tmp/measure-projection.out on the dev box.

Flag state reminder for the takeover

• DKG_LIST_CONTEXT_GRAPHS_PROJECTION defaults OFF — C3's projection-backed listContextGraphs is dead code until enabled; merging alone changes nothing for the enumeration.
• The read-both collapsed-UAL delta arm was re-ported into A1's durableDeltaWhereClauseForGraphs (graph-plan.ts) but is inert until A5 lands the DKG_SYNC_DELTA activation — correct and ready, just not exercisable on this branch.

🤖 Generated with Claude Code

[branarakic]

Closing as redundant — Track-C reconciled into main via #1201 (integration/track-c-merge).

Verified at content level (not just patch-id): this branch is 169 commits behind main; 6/7 of its distinctive SWM-admission identifiers exist in main (createResponderSubGraphRegistrationMemo, createSubGraphNameMemo, filterSharedMemoryMetaQuads, isSharedMemoryBucketDescendantDataGraph, effectiveRegisteredSubGraphNames, subGraphRegistrationMemo), and the 7th — the "honor child CG collisions in SWM admission" fix (3795aec11) — is present in main under renamed identifiers (graph-plan.ts:643: childCgUri + isKnownContextGraph, the same child-CG-collision skip logic). No unique unmerged work found. Reopen if I've missed something.