Per-KA metadata trim: ~134 → ~50 quads/KA (RFC + Phases 0–3 implementation)#1155
Conversation
One 1-triple KA publish leaves ~134 resident quads (live-measured); ~97% is bookkeeping, ~30 quads are copies of five values. Combined with hot-path graph-name scans (STRSTARTS(STR(?g)), SELECT DISTINCT ?g — the adapter's own "dominant idle-node CPU cost"), this drives the rc.17 idle-CPU saturation. The RFC specifies: Phase 0 dead code, Phase 1 zero-reader drops (~-24/KA), Phase 2 dedupe via small reader migrations (~-25), Phase 3 aggressive decision points (UAL/token collapse, URN merge, provenance-events flag, ~45-50/KA), and the query-side fixes (graph registry to kill name scans, event-driven reconcilers). Every verdict cites its writers and readers by file:line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements docs/rfcs/ka-metadata-trim.md: one 1-triple KA publish drops from ~134 resident quads to ~50 (~40 with metadata.provenanceEvents=false), with every removed/relocated triple justified by a writer+reader audit and every migrated reader reading both old and new shapes. - Phase 0: dead AssertionPublished writer+gate; orphan kcUal read; dangling partition bnode. - Phase 1: zero-reader drops (kaCount, blockTimestamp, publisherAddress, chainId, blockNumber, tokenId, publicTripleCount, AuthorshipProof block, Publication node, URN type/contextGraph/wasGeneratedBy rows). publishedAt KEPT after adversarial review found the kafka-plugin discovery reader. - Phase 2: KC/KA type rows -> predicate-based counters; single rootEntity (entity alias dropped outside the signed seal); wm/swm pointers written only on divergence; fromLayer/toLayer + wasAssociatedWith derived/optional; publicSnapshotRef collapsed; WM marker updated (not deleted) at VM flip. - Phase 3: UAL+<ual>/<n> collapsed to one node (read-both in resolveKA, access-handler incl. <ual>/<n> fallback for old clients, RS prover, sync, kafka discovery, async-lift, EPCIS, counters); metadata.provenanceEvents config (default true) gating all four lifecycle event writers; ShareTransition dropped (node-ui receipt reads the seal subject, legacy fallback retained); partition CONSTRUCT-copy -> documented minimal shape (REMAP keeps wholesale move). Lifecycle-URN->seal merge DEFERRED with a worked plan (TODO(rfc-ka-trim) at assertionLifecycleUri) after the audit surfaced signed-material collision + identity double-allocation hazards. - Adversarial-review fixes: kafka read-both queries; provenanceEvents gating on discard/update; multi-root private access attestation always matches the served triples (computePrivateRoot fallback); stale re-promote stays a no-op; isAlreadyConfirmed read-both vs the minimal partition shape. Author-signed seal material untouched. Suites green: core 1039, query 261, random-sampling 62, publisher 1150, agent 1226, cli 2031, kafka-plugin 169, node-ui 1456 (+tsc/builds). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| if (q.predicate === `${DKG_NS}merkleRoot`) kcMerkleRoots.set(q.subject, stripLiteral(q.object)); | ||
| } | ||
|
|
||
| // TODO(rfc-ka-trim) P3.1: collapsed-shape rows (rootEntity on the UAL, no |
There was a problem hiding this comment.
🔴 Bug: this leaves collapsed-shape KAs (rootEntity on the bare UAL, which this PR now writes by default) on the existing "accept on trust" path, so sync no longer Merkle-verifies newly published metadata at all. That is a silent data-integrity downgrade for replicated stores. Either teach the verifier to self-map bare UAL subjects before enabling P3.1 writes, or keep emitting the legacy partOf rows until the verifier is upgraded.
| ) | ||
| OPTIONAL { ?asrt dkg:publishedAtKaId ?kaId . } | ||
| OPTIONAL { ?asrt dkg:assertionFinalizedAt ?finalizedAt . } | ||
| OPTIONAL { ?lc dkg:rootEntity ?asrt ; dkg:reservedUal ?ual . } |
There was a problem hiding this comment.
🔴 Bug: ?lc dkg:rootEntity ?asrt does not match the current lifecycle shape — those dkg:rootEntity rows point at member entities, not the assertion URI. After this PR stops writing publishedAtKaId, that means ?ual/?batchId never bind for new publishes, so the hook will return status: 'verified' with ual and kaId still null. Join through a real assertion→lifecycle edge, or keep a direct id field until that link exists.
| // old-client requests keep resolving against new-shape stores. | ||
| const legacyToken = /^(.+)\/\d+$/.exec(kaUal); | ||
| if (legacyToken && legacyToken[1]) { | ||
| return this.queryKAMeta(legacyToken[1]); |
There was a problem hiding this comment.
🔴 Bug: collapsing every legacy <ual>/<n> request to the bare UAL loses per-root addressability for multi-root KAs. Once queryKAMeta() runs on the collapsed shape, both /1 and /2 resolve to whichever rootEntity happens to be first, so non-first private bags become unreachable and callers can receive the wrong member's data. This needs an explicit root-selection mapping (or retained token metadata) before falling back to the bare UAL.
| mq(subject, `${RDF}type`, `${DKG}Assertion`, metaGraph), | ||
| // Assertion entity (DKG identity). RFC ka-metadata-trim Phase 1: the | ||
| // `a prov:Entity` / `a dkg:Assertion` type rows, `dkg:contextGraph` and | ||
| // `prov:wasGeneratedBy` were dropped (zero readers — history joins the |
There was a problem hiding this comment.
🟡 Issue: prov:wasGeneratedBy still has a live in-repo consumer: packages/graph-viz/src/core/provenance-resolver.ts reads it to populate generatedBy / generatedByName. Dropping the triple here will make newly created assertions lose provenance in graph-viz. Either update that consumer in this PR or defer removing the edge.
- sync-verify: collapsed-shape KCs now Merkle-verified on sync (self-map in both duplicated impls, multi-map roots, dual-shape dedupe guard) — a wrong merkle root on a collapsed KC is now rejected instead of accepted on trust. - node-ui receipt: replaced the never-binding lifecycle join with the member-entity join pinned by the URN/URI tail correspondence; the identical latent bug in the legacy hop fixed too; real executed-query tests added. - multi-root access: conditional collapse — multi-root KAs re-emit per-token pairing rows (manifest order) alongside the collapsed shape, so legacy <ual>/<n> resolves exactly root N with matching attestation; single-root publishes (dominant case) keep the full collapse. Handler serves the first root that has a private bag; F3 recompute guard retained. - graph-viz wasGeneratedBy: documentation-only correction (generic matcher, no stranded feature) — RFC + inline comment amended. Tests: agent unit + 9 sync-path hardhat suites green; publisher full 1232; node-ui full 1423 (+ new executed-query tests); graph-viz 140; builds green. Full agent lane delegated to CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
All four review comments verified independently against the code, then fixed in
Suites: publisher full 1232 ✅, node-ui full 1423 ✅ (incl. new executed-query tests), agent unit + all 9 sync-path hardhat suites ✅, graph-viz 140 ✅. Full agent lane delegated to CI on this push. |
| ?ka <${PART_OF}> <${ual}> . | ||
| { ?ka <${PART_OF}> <${ual}> . ?ka <${ROOT_ENTITY}> ?root . } | ||
| UNION | ||
| { <${ual}> <${ROOT_ENTITY}> ?root . } |
There was a problem hiding this comment.
This single-UAL query needs the same dedupe as the list path. Pre-trim rows can satisfy both UNION arms because the bare UAL carried aggregate dkg:rootEntity rows while <ual>/<n> also carried dkg:partOf + rootEntity. buildListQuery hides that with SELECT DISTINCT ?ual ?root ?receivedAt, but here every data triple is returned twice, and bindingsToKa() turns scalar properties into duplicate arrays (e.g. schema:name: ["demo", "demo"]) after handleGetSingle adds the UAL back. Please make this SELECT DISTINCT ?root ?p ?o or dedupe before bindingsToKa().
| OPTIONAL { | ||
| ?lc dkg:rootEntity <${entityIri}> ; | ||
| dkg:reservedUal ?ual . | ||
| FILTER(STRENDS(STR(?lc), CONCAT(":", REPLACE(STRAFTER(STR(?asrt), "/assertion/"), "/", ":")))) |
There was a problem hiding this comment.
This only pins the lifecycle row by {addr,name}. For a sub-graph assertion such as did:dkg:context-graph:cg/demo/assertion/0x.../name, STRAFTER(..., "/assertion/") drops the demo segment, so this filter also matches urn:dkg:assertion:cg:0x...:name or urn:dkg:assertion:cg:other:0x...:name whenever those lifecycle rows stamp the same member entity. That gives the UI the wrong reservedUal/agent/batchId for same-named root/sub-graph assertions. The new tests cover a different assertion name, but not the same-name/different-subgraph case. Please derive the expected lifecycle URI from the full assertion scope before /assertion/ (including optional subGraphName) before binding ?ual.
Coordination note: this PR is the data-volume half of the #1138 scalability waveFor reviewers / whoever sequences these into A semantic cross-check across the wave confirmed nothing reads a
Everything else in the wave is clean rebase friction or complementary build-out. |
|
Agreed with the coordination framing — #1155 (quads-per-graph) and the Proposed sequence (so the wave rebases once, against the final data shape):
One scoping clarification so nothing falls between the two efforts: validating A1 (#1141) against the production oxigraph-server backend (not the in-memory embedded store its tests use) showed the durable-meta page is still O(meta×lifecycle) on the server — the per-row 3-arm Coordination points confirmed (whoever rebases second owns): (1) re-port the collapsed-shape arm into A1's |
The two ka-extractor regression-lock tests asserted a FULL-collapse shape (zero <ual>/N token rows) that #1155 abandoned. Root cause is a two-commit contradiction within #1155: 369042c rewrote source+tests to full collapse, then the Codex-review follow-up 09fc5ec added the CONDITIONAL collapse back to the source (metadata.ts:900-919 — multi-root re-emits per-token rows in manifest order alongside the collapsed UAL-subject set) but left these two tests at the interim expectation. The source is correct and load-bearing (required by the green multi-root-token-rows suite + AccessHandler per-root private-bag resolution); only the tests were stale. Restored the GENUINE invariants against the actual conditional-collapse shape — not weakened: - bug 2 (manifest order, N=12): assert <ual>/i dkg:rootEntity urn:new:i AND partOf <ual> for i in 1..12 (one binding each) — catches a lex-sort regression (<ual>/10 -> urn:new:2). Kept the collapsed UAL-subject set check. - bug 3 (surplus deletion, 5->2 shrink): assert exactly {<ual>/1,<ual>/2} survive as partOf tokens, <ual>/3..5 are fully gone (the real surplus lock), tokens bind urn:new:a/b in manifest order, UAL subject = {urn:new:a,urn:new:b}. Verified: both restored asserts proven to FAIL under source mutation (lex-sort -> bug2 red; drop surplus-delete -> bug3 red). Single-root collapse and merkle leaves unaffected (leaves key on distinct root URI, not token). No source changed. random-sampling 62/62 green (clean rebuild). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI blocker fixed:
|
…h scope Addresses the two open review threads on PR #1155 (both read-both / collapsed-shape correctness): 1. kafka-plugin discovery single-UAL query (discovery.ts): on a pre-trim / mixed-version store both meta-block UNION arms (legacy <ual>/<n> partOf token + collapsed bare-UAL rootEntity) bind the same ?root, returning every data triple twice -> bindingsToKa produced duplicate-array scalars (schema:name: ["demo","demo"]). Add SELECT DISTINCT, mirroring buildListQuery. 2. node-ui receipt lifecycle filter (useEntityOnChainReceipt.ts seal query): pinned the lifecycle URN by only :{addr}:{name}, so a same-named assertion in a sibling sub-graph leaked the wrong reservedUal/agent/batchId. Pin to the FULL assertion scope. cgId is a build-time constant, so anchor on it as a literal (preserving slashes — wallet-scoped cgIds like <addr>/<name> embed RAW in the URN via assertionLifecycleUri) and only '/'->':' convert the [/sub]/{addr}/{name} tail derived from ?asrt. Regression tests (proven red-without-fix): - kafka: single-UAL builder asserts SELECT DISTINCT + bindingsToKa scalar lock. - node-ui: real-store receipt-subgraph-scope.test.ts — cross-subgraph (red under :{addr}:{name}-only) AND wallet-scoped slash-cgId (red under a blanket '/'->':' that rewrites the cgId's own slash); root + sub-graph, no-slash + slash. kafka discovery 49/49, node-ui receipt 9/9 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both open review threads addressed —
|
…1138 Brings the data-volume half (#1155) under the A1+A2 access-pattern work. Single conflict: sync-handler.ts durable-data else branch — took A1's structure (inline delta query was re-homed to graph-plan.readDurableDataPage); #1155's read-both collapsed-KA arm is already present in A1's durableDeltaWhereClauseForGraphs (legacy partOf arm + collapsed rootEntity+batchId arm), so the coordination re-port is satisfied. #1155 preserves the sync _meta predicates (memoryLayer/assertionGraph), so A1 durable-meta admission + D-SEC are unaffected. Builds green (tsc 8/8).
…hygiene (#1143) Squash-merge of codex/issue-1138-a3-sync-scheduler-progress (15 commits) onto integration/issue-1138 (A1+A2+#1155), plus one stale-test fix. A3 makes the sync requester/scheduler honest about failure: - separates peer-reachability failures (failedPeers) from phase failures (failedPhases); the lifecycle success-gate requires both zero, so a failed/timed-out/denied round no longer stamps success → backoff engages - per-(peer, contextGraph) progress accounting (defines a 'clean round') - freshness-aware durable checkpoints; metadata-only freshness is not counted as data progress - flap hygiene: cooldowns/backoff survive connection:close Validation (worktree dkg-a1-verify, integration is not CI-gated): - build + tsc clean (agent/cli/node-ui) - agent suite 1361 passed / 0 failed; node-ui 1431/0 - cli 82 fails are pre-existing Windows-host env (identical on base) — A3's own new cli tests pass - server-backed responder perf guard 3/3 (A3 doesn't touch the responder) - 4-lens adversarial review: 0 confirmed issues Stale-test fix: swm-snapshot-sync 'remote snapshots unavailable' is a phase failure, not a peer failure → assert failedPeers=0 + failedPhases=1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 participants
What
Implements RFC: per-KA metadata trim + indexed graph addressing (in this PR), Phases 0–3.
Publishing one 1-triple KA currently leaves ~134 resident quads in the local store (live-measured on a Base Sepolia rc.17 node) — ~97% publish bookkeeping, ~30 quads repeated copies of five values. That volume (× network sync) feeds RocksDB compaction and makes every recurring reconciler query more expensive — the mechanism behind the rc.17 idle-node CPU saturation. After this PR: ~50 quads/KA, ~40 with
metadata.provenanceEvents: false(lite mode).How
Every removed/relocated triple was justified by a writer + reader audit (grep both IRI forms over all packages incl. node-ui/mcp-dkg; verdicts cited per-row in the RFC). Every migrated reader reads both old and new shapes (replicas hold old-shape rows synced from older nodes).
AssertionPublishedwriter+gate, orphankcUalread, dangling partition bnode.kaCount,blockTimestamp,publisherAddress,chainId,blockNumber,tokenId×2,publicTripleCount×2, AuthorshipProof block, Publication node, lifecycle-URN type/contextGraph/wasGeneratedByrows.rdf:typerows → predicate-based counters; onerootEntity(thedkg:entitydual-write dropped outside the signed seal);wm/swmpointers written only on divergence fromvm;fromLayer/toLayerderived from event class;wasAssociatedWithoptional;publicSnapshotRefcollapsed into the digest.UAL+<ual>/<n>collapsed into one node (post-rc.17 invariant: 1 publish = 1 KA = 1 UAL);metadata.provenanceEventsconfig (defaulttrue) gating all four lifecycle event writers; ShareTransition removed (node-ui receipt reads the seal subject, legacy fallback retained); partition copy → documented minimal shape (REMAP keeps the wholesale move — it's the only meta home there).Deliberately deferred (with worked plan)
Lifecycle-URN → seal-subject merge: the implementation audit surfaced (a) subject collision with author-signed seal/receipt rows under subject-scoped wipes, and (b) exact-subject
kaId/reservedUalreads risking identity double-allocation on upgraded stores. Plan atTODO(rfc-ka-trim)inpackages/core/src/constants.ts.Verification
partOf+publishedAt(audit miss —publishedAtis now KEPT, RFC corrected) → read-both queries; the kafka e2e reproduced the bug live pre-fix (red→green).provenanceEvents=falsedidn't gate discard/update event writers → gated,prov:wasInvalidatedBygated with its event."VM"instead.isAlreadyConfirmedASK vs the minimal partition shape → read-both against label_meta.Reviewer focus, please
<ual>/<n>→bare-UAL fallback, RS prover, sync delta, kafka) — highest blast radius.dkg:partOfwrite removal (RFC "Invariant exception" section) — sanctioned, all readers read-both, but it's a model change worth a deliberate ACK.metadata.provenanceEventsdefault (currentlytrue= zero behavior change; flipping tofalsemaximizes savings network-wide).🤖 Generated with Claude Code