feat(manager,core,cli)!: move the control surface onto the v0.4 endpoint rails - #285
Merged
Conversation
…llocation
P2 slice 1, Unit A (the freeSlot-split file/allocation half; the supervised-restart
rail and crash-durable reconciliation land in Unit B behind a SPEC amendment).
Closes two teardown/allocation defects:
- Manager-provisioned incarnations now embed their lifecycleUid in the on-disk
secret family (`<name>-<uid>.creds`, actor-token, sentinel, health), mirroring the
broker-side name-disjointness. ManagedAgent records the exact family at spawn and
at resume-adoption; teardown, preservation, and the health read consume the
recorded-or-uid-derived paths only, never a re-derivation by name. A stale or
replayed teardown for a retired incarnation can no longer delete a same-alias
successor's files or a name-keyed standing operator credential it holds no record
of. New workspace builders agentLifecycle{SecretFilePaths,CredsKey,...} and
agentSecretKeyForFile (derives the store key from a recorded path). The manifest
ledger records the spawn reply's lifecycleUid so `down -f` derives the same
lifecycle-keyed path; rows without it fall back to the legacy name-keyed layout.
- uniqueName consults the live presence roster (status !== "offline") in addition to
the managed/reserved/retiring sets, awaiting the initial presence snapshot first,
so a spawn against a name a live unmanaged peer already holds auto-numbers instead
of minting a doomed sibling the broker refuses (the 30s launch-uncertain hang).
Presence is advisory: this is an availability optimization at allocation; the
broker/auth remain authoritative. Offline rows never occupy, so a retired name
stays reusable.
New smoke:lifecycle-files proves both by execution: lifecycle-keyed cred file,
successor + operator creds survive a replayed retired-incarnation teardown, live
same-name peer auto-numbers, offline row stays reusable.
…spaces The per-incarnation filename base flattened (name, uid) to `<name>-<uid>`, but `-` is a legal character in a standing agent name: a standing alias literally named `worker-<uid>` shared the exact store key and file path with the lifecycle family of alias `worker` at that uid, so a lifecycle teardown could delete a foreign standing credential (cross-alias clobber, the same class the lifecycle keying exists to close). Entropy in the uid is no defense: a standing alias is chosen, not sampled. The incarnation base is now `<name>.<uid>` with a `.` separator, a character the standing-name alphabet refuses, so no legal standing name can spell an incarnation base. The provisionable-filename grammar (recorded-path key derivation and the clean sweep) accepts exactly the two builder families and nothing else. Also folded from review: - The spawn allocation's presence-snapshot await is now unconditional so a half-wired endpoint fails loud instead of silently allocating off a pre-snapshot roster; the fake-endpoint smokes stub the seam explicitly. - A managed slot with nothing recorded keeps `secretPaths` absent (never an empty object), so teardown's uid-keyed derivation fallback stays reachable. - The ownership smoke now proves the exact colliding-alias negative (standing `worker-<uid>` survives the lifecycle teardown of (worker, uid), recorded-path and uid-derive replays both) and proves the snapshot-before-allocation ordering with a deferred snapshot gate (mutation-verified: removing the await fails the test). - Migrated the name-keyed cred-path assertions in the manager smokes (lifecycle-e2e, persona-identity-acl, manifest-launch, transcript-grant-acl) to the lifecycle-keyed layout.
A retained user-mode inventory pinned its actor-token and sentinel paths
independently (each accepted as the lifecycle-<uid> OR the legacy
name-keyed path), and never pinned the health path at all. A corrupt or
admin-supplied inventory could therefore mix families and, worse, point
`health.path` at an arbitrary workspace file: that path flowed into the
bearer argv and was rmSync'd at terminal teardown. Reproduced at the
prior hash (resume accepted a foreign health path, the child's argv
carried it, the terminal cleanup deleted the victim file).
The resume now pins all three of {actorToken, sentinelCreds, health} as
ONE family: the recorded triple must equal the full lifecycle-<uid>
triple or the full legacy triple. Mixed families and foreign paths are
refused before any spawn. Health is pinned by path equality, never by
file existence, so a transiently-absent health file still validates.
preserve-state adds the user-mode negatives: a foreign health path is
refused before any spawn and the foreign file survives; a mixed family
(lifecycle token + legacy sentinel/health) is refused.
Also migrates the manager-spawn cred-path assertions in the user-mode
live smokes to the lifecycle-keyed layout (`<name>.<uid>.<kind>`), each
recovering the incarnation uid from disk:
- down-manifest-usermode (a pnpm check lane): green, 25/25.
- user-spawn: green through every migrated assertion.
- freeslot-respawn-barrier, int2-revoke-hold: the single token-read site
each.
Move the lifecycle head/uid/gate key builders, the closed LifecycleMapping and EpGateRow value schemas with their boundary parsers, and the state constants from implementations/auth/src/lifecycle-registry.ts into the new packages/core/src/lifecycle-state.ts, so every lifecycle executor shares one encoder instead of growing per-executor copies. The registry now consumes the core grammar (types re-exported, package surface unchanged) and the two inline records-bucket names now use the core recordsBucket builder, retiring a second un-tokenized encoding of that name. Behavior-neutral: no sequencing, orchestration, or grant changes. Proof: lifecycle-registry 79/79, retirement-barrier 61/61, auth-admin 14/14, credential-ledger 78/78; pnpm typecheck green.
…ected transport Split #2 of the panel-locked three-way split (design note @ dc8d1b8). The activation saga (reserve -> gate frozen -> head CAS -> reopen LAST) with its crash-resume, the gate CAS transitions, the uid reservation, and the barrier-internal head steps (epoch advance + root stamp + retirement begin/complete) move to packages/core/src/lifecycle-saga.ts as ONE shared state machine over an injected LifecycleStateTransport. The sealed registry now implements that transport over its own internals (the WeakMap brand still gates every write) and every public function delegates with its signature, refusal matrix, and error faces unchanged; a second executor (the manager's static adapter) can now drive the SAME saga instead of a second copy of a security-critical CAS sequence. Behavior-neutral for user mode by execution proof: lifecycle-registry 79/79, retirement-barrier 61/61, auth-admin 14/14, credential-ledger 78/78, barrier-plane 60/60 (incl. boot crash-resume), full smoke:ci green, typecheck green.
Third grammar tranche of the three-way split (design note @ dc8d1b8, B1): the cred./epcred. family keys, the closed CredentialLedgerRow schema with its per-family key-binding parser, and the segment/credential-id/holder/ source-chain asserts move from implementations/auth/src/credential-ledger.ts into packages/core/src/lifecycle-state.ts. The ledger imports and re-exports them (package surface unchanged), so the manager's static lifecycle adapter can write and revoke its B1 ledger rows in the ONE encoding the auth ledger reads, instead of a second copy of the row grammar. Behavior-neutral by execution proof: credential-ledger 78/78, lifecycle-registry 79/79, retirement-barrier 61/61, auth-admin 14/14, barrier-plane 60/60; typecheck green. (int2-revoke-hold currently fails identically with and without this change: a pre-existing branch issue in the secret-store prepareServer path, tracked separately.)
Static meshes now run the same lifecycle registry user mode runs, over the shared core saga: - Activation: every static spawn persists a durable outer intent (the new records-store slot row mgrslot.<owner>.<alias>, phase provisioning) BEFORE driving the shared activation saga at its uid. The wire authority principal is the incarnation-unique nkey; the alias is routing only, protected by the name-keyed slot mapping + uid reservation + the manager's retiring hold (the head is principal-keyed and does not serialize names). - Terminal: despawn drives the static retirement barrier in the normative order - gate freeze -> head retiring -> ledger revoke (all rows, incl. renewals) -> footprint cleanup inside the barrier -> gate retired -> head retired -> slot retired -> alias free. Recovery is exact-op: a same-op terminal gate resumes to the head; a crashed activation is finished or burned through its own op. - Credentials: manager-spawned static agent creds are bounded (24h) and ledgered (cred.<uid>.<credId>) before materialization; the manager is the renewal owner (push remint for live slots; a child JWT is never proof of incarnation; nothing mints after terminalizing). Resume recovers the nkey seed so adopted agents stay renewable. - Control: manager control ops are authorized against the AUTHENTICATED incarnation principal - a live managed slot passes, a terminalizing/ retiring/retired incarnation's credential is refused even with a tier-valid JWT, and unknown principals keep plain tier authority. A terminalizing latch flips before the first await on every stop path. - Reconciliation: manager start (under the lease) sweeps the durable slot rows and re-drives crashed activations/terminals, so no active orphan survives a restart. cotal up now seeds the two authority stores on every auth mesh, and a new key-pinned one-shot lifecycle-executor profile scopes every state write to exactly one incarnation's keys. - Static spawns refuse endpointCapabilities fail-closed (the static terminal has no obligation-drain/frontier steps yet). New smoke:static-lifecycle (32 checks, live broker) covers activation state, the membership gate, renewal and its terminal refusal, the terminal order, same-name respawn, crash rollback, reconciliation, and the capability refusal. Docs: security.md/roadmap.md updated to the new credential story; docs bundle regenerated.
The boot reconcile deferred every active slot while a resume was pending (it could not yet know which would be adopted), but nothing re-swept after finalizeResume - so an active slot the resume did NOT adopt (a crash after slot->active but before agents.set, with the rest of the fleet preserved and resumed) survived forever. That wedged its alias against reuse AND left its principal out of the retirement refusal index, so a copied credential of that incarnation fell through to plain tier authority until its TTL - reopening the F5(a) control-plane residual on the resume path. - reconcile now decides orphan-vs-live by genuine membership (this.agents at the same uid), never the blanket resumeRequired flag. - the boot sweep still DEFERS active slots while a resume is pending (adoption runs after it); provisioning/terminalizing never defer. - finalizeResume runs a post-adoption sweep (postAdoption=true) while resumeRequired still blocks ordinary spawns, so this.agents is exactly the adopted set - any active slot not adopted terminalizes and its principal enters retiredPrincipals before ordinary control serves. smoke:static-lifecycle gains 4 checks reproducing the exact scenario: boot-defer, still-unrefused-while-pending, post-adoption terminal, and the now-closed F5(a) refusal - plus the live agent surviving both sweeps. Found by the review panel (distsys + security) on the 9e13648 hash.
Panel LOW folds on the Unit B static executor (non-blocking, all three reviewers APPROVEd at a478f53; folded for guarded-core hygiene): - lifecycle-executor no longer trusts a caller-supplied slotKey literal (previously only regex-validated, so a mis-constructed pin could name a foreign records key for the slot write - claim-exceeds-enforcement). The profile now takes `alias` and derives staticSlotKey(owner, alias) internally, exactly like the head/uid/gate keys, so every key is coherent by construction and the "one incarnation" promise is enforced, not asserted. - provisioner grant comment corrected: the reconcile sweep DOES read slot bodies (phase/uid/actor) to plan resume, via the stream-scoped MSG.GET residual - the prior "no bodies delivered" clause was false. - reconcileStaticLifecycles JSDoc updated to the genuine-membership adoption predicate + the boot-defer / post-adoption two-sweep model (was still describing the old blanket-resumeRequired behavior). Behavior-neutral: smoke:static-lifecycle stays 37/37, typecheck clean.
The §13.7 cluster document + compiled command contracts that let the manager register as a v0.4 `service` endpoint and serve typed commands on the ep.* rails. This is the pure-data / schema half of slice 1a (no broker, no barrier, no wire I/O): the walking skeleton declares ONE read-only command `status` (manager-level health: instanceId/runtime/agentCount/uptimeMs), void-input, with a real compiled output schema. 1b appends the remaining ops (start/stop/attach/definePersona/purge/models/launch + resume) as further commands in this document. Self-consistency verified (7 checks): the document's pinned digests match the compiled contracts, the two-digest §13.7 manifest/root verify, and the compiled output schema validates a well-formed status and rejects a missing field. The security-critical serve WIRING (contract-store publication, the §13.1 issuance barrier, the endpoint-serve mint, serveEndpoint) is the next 1a increment, carrying the panel's 1a build checklist.
…1a-gate) Move the §13.1 ENDPOINT-instance issuance-gate grammar (epgate.<endpoint>. <instanceId>): the EndpointGateRow schema, epgateKey, and the boundary parser parseEndpointGate — from implementations/auth's session-ledger to @cotal-ai/core (lifecycle-state.ts, alongside the epcred family it pairs with). Behavior-neutral, byte-identical parser; auth imports + re-exports for an unchanged package surface. Reason (panel-blessed, fact H3): the endpoint-serve credential subsystem is core's own deferred post-D14 wiring, and P2 item 1 makes the manager its first production consumer. The manager cannot import implementations/auth (AGENTS.md one-way deps), so the shared grammar must live in core — the same guarded-core lift as the Unit B lifecycle-state/saga lifts, closing the dual-encoder drift a manager-local epgate dialect would open. The gate/barrier builders follow in the next 1a-gate commits. Verified: core + auth typecheck clean; the session-ledger smoke stays green (81/81, incl. the epgate-grammar-parity and D14 serve-issuance-gate sections).
…1, 1a-gate) Move the two generic raw-KV credential-ledger row helpers createRowByteIdempotent (create-only byte-idempotent stage) and markLedgerRowRevoked (revision-pinned active->revoked CAS) from implementations/auth's credential-ledger into a new core module endpoint-serve-kv.ts — the shared home for the §13.1 endpoint-serve credential lifecycle over a plain KV. Both take a bound KV and use only core primitives (parseLedgerRow, isCasLoss); no auth-store dependency. auth imports them for its own writes and re-exports them, so its package surface is unchanged and all four internal callers (root-credential, retirement-barrier, session-ledger, credential-ledger) keep working. Reason (panel-blessed 1a-gate): the manager's endpoint-serve wiring needs these primitives and cannot import implementations/auth (AGENTS.md one-way deps), so the KV binding lives in core — the guarded-core lift that lets the mint fence and the production issuance barrier (next 1a-gate commits) share one implementation. Behavior-neutral: full typecheck clean; session-ledger 81/81 and credential-ledger 78/78 stay green.
… 1, 1a-gate)
Move the endpoint-serve mint-fence logic (the EpIssuanceGate observe/stage/
commit/revoke over the epgate.<endpoint>.<instanceId> + epcred.<…> families)
from implementations/auth's kvServeIssuanceGate into core endpoint-serve-kv.ts
as serveIssuanceGateKv(kv, space, {endpoint, instanceId}). All its deps are now
core (the lifted epgate grammar + ledger-row primitives, parseLedgerRow,
isCasLoss). auth's kvServeIssuanceGate keeps its branded SessionAuthStore, only
unwrapping it to (kv, space) and delegating — the space bond stays constructed,
not asserted.
Reason (panel-blessed 1a-gate): the manager mints its endpoint-serve credential
through the SAME fence and cannot import implementations/auth (AGENTS.md one-way
deps). This is the last lift before the net-new production EpIssuanceBarrier +
epgate provisioning. Behavior-neutral: full typecheck clean; the session-ledger
smoke stays green (81/81, incl. the D14 serve-issuance-gate section that drives
this fence over the real KV).
…oning (P2 item 1, 1a-gate) Add the FIRST production wiring of the endpoint-serve credential subsystem (fact H3) to core endpoint-serve-kv.ts: - provisionEndpointGateOpen: create-only births the epgate.<endpoint>. <instanceId> gate OPEN at generation 0 bound to the serving principal (the §13.1 pre-registration a registerServiceInstance writes behind); idempotent for the same principal, conflict for a different one (an instance token is never re-bound). - endpointRegistrationBarrier: the production EpIssuanceBarrier over a plain KV that registerServiceInstance drives — observe / freeze (open->frozen under a registration op, returning the fencing token) / enumerate (a REAL epcred prefix scan) / revoke (active->revoked CAS per row) / evict (injected; the trivial fresh-registration evictor is a NAMED residual, sound only with no live predecessor) / reopen (token-pinned frozen->open at the successor generation). The freeze/reopen CAS is the real fence: a moved gate makes a racing mint LOSE. This is critic's "no seed shortcut": serve authority moves ONLY through the gate CAS, never a seed-signed shortcut. New broker smoke smoke:ep-serve-gate (22 checks, in smoke:ci): provision the gate open, drive the barrier's observe->freeze->reopen gate-CAS traversal, mint through serveIssuanceGateKv (stage epcred + revision-pinned commit; a stale commit loses), a takeover barrier enumerates + revokes the prior serve family (the real revoke path, non-reissuable under the old generation), and the loss negatives (freeze of a frozen gate loses, a stale-token reopen loses, a different-principal re-provision conflicts). Proves the panel's 1a-gate musts 1/3/6 over a real broker + real auth KV.
…arrier
The registration barrier's default evictor was `() => true`, which the review
panel showed is a silent fail-open: the saga's takeover guard is
`if (!(await barrier.evict(holderPrincipal))) throw "not verified evicted"`
(endpoint-service.ts:524-526), so `() => true` makes `!true === false` → no
throw → a takeover with a live predecessor and no injected real evictor revokes
the prior serve family and REOPENS at the successor generation WITHOUT ever
verifying the predecessor's live connection gone — split-brain serving until
TTL. The default is only ever consulted on a NON-EMPTY family (a takeover; a
fresh registration's family is empty), so "sound for fresh registration" was
vacuous and the value's only effect was a false "verified GONE" on takeover.
Default is now fail-closed `() => false` ("no evictor ⇒ eviction cannot be
verified ⇒ not verified"): a takeover with no injected real evictor fails closed
(the saga leaves the gate frozen for reconciliation, its own intent), fresh
registration is unchanged (evict never invoked), and the real $SYS CONNZ+KICK
evictor is still injected by a caller that has it. This converts the named
residual into the enforced guard the saga already claims.
smoke:ep-serve-gate updated (24 checks): section D now proves the default
reports NOT-verified on a non-empty family and injects a test evictor for the
happy takeover path (the prior smoke asserted the trivial default `=== true`,
ratifying the hole); plus a belt check that a mint commit while the gate is
frozen loses. Found by the panel at d78eacb.
…1, 1a-serve) Add the one-shot `endpoint-serve-executor` credential profile — the barrier connection identity the panel pinned for 1a-serve (critic #1). It grants WRITE to exactly ONE (endpoint, instanceId)'s endpoint-serve state: the gate `epgate.<ep>.<iid>` (the registration barrier's provision + freeze/reopen CAS) and the serving ledger family `epcred.<ep>.<iid>.>` (the mint fence's stage + the barrier's revoke). Every key is derived inside the profile from (endpoint, instanceId) via the core builders (new epcredFamilyPrefix), never a caller literal, so a leaked/mis-constructed executor moves exactly one endpoint instance's serve state and nothing else. Reads are leader-served MSG.GET on the auth store (allow_direct=false; stream-scoped named residual, no bearer bytes) plus the ordered consumer the epcred keys() enumeration binds. This lets the manager (1a-serve) drive the registration barrier's gate CAS + the serve mint THROUGH this scoped executor rather than its standing seed/supervisor connection — the manager-specific half of "no seed shortcut" the throwaway 1a-gate smoke could not discharge. Verified: grants are exactly key-pinned (no broad epgate.>/epcred.> wildcard, no records-store write, inbox-scoped sub, a non-KV-safe pin refused by the builder); full typecheck clean; cred-lifetime (18) + frozen-exports (7) profile-matrix smokes green.
…atus` (P2 item 1, 1a-serve) The manager now traverses the SAME endpoint-serve credential path an ordinary v0.4 service uses, over a real broker, at start(): - provision the SPEC 13.1 issuance gate at (space, "manager", managerLifecycleUid), bound to the serve principal minted for it; - drive registerServiceInstance through the endpoint registration barrier's gate CAS (freeze -> authorize under freeze -> publish -> reopen at the successor), publishing the 13.7 cluster document from an in-memory content store (the manager authors it; the epc store publication joins the D8 loader slice); - authorize the serve grant with the processEpoch read from the GATE (never derived from the uid) and mint the endpoint-serve credential behind the release fence's revision-pinned CAS win. Every credential write rides a one-shot, key-pinned endpoint-serve-executor connection - never the standing supervisor connection. The executor profile gains the registration's two records-store keys (the instance's svc spec + the endpoint governance head) so the whole registration is executor-scoped; the standing supervisor cred remains broker-denied on all of them. serveEndpoint(status) then dual-serves beside the legacy ctl tiers on a dedicated connection whose authenticator always presents the CURRENT credential, and both dispatch doors now run ONE shared admission chokepoint (admitControl): the maintenance/resume fence plus the F5(a) lifecycle-membership gate, mapped onto the 13.3 catalog on the ep door (unavailable / permission-denied). The standing renewal pass re-mints the serve credential through the same fence (a distinct ledger row per JWT, same stable nkey); stop() and the lease-loss exit tear the serve loop down. New manager-service smoke (in smoke:ci) proves the binding 1a checklist over a real Manager + JWT broker: gate-before-register, the no-seed-shortcut traversal (positive ledger proof + the supervisor denial negative), the shared chokepoint on both doors (retired principal and resume-pending, both restoring), epoch/uid disjointness, typed epCall(one) with a gate-backed currency reader, describe, ctl dual-serve, fence-traversing renewal, and stop teardown. Full typecheck + smoke:ci green; lifecycle-e2e 30/30.
…s (P2 item 1, 1b) The manager's rev-2 cluster document now serves EVERY control op as a typed command on the ep rails, dual-served beside the untouched ctl tiers: status, ps, inspect (the per-agent read; renamed from the ctl named "status" to keep the 1a manager-level status distinct), models (normalized to a catalogs list), spawn (the full 16-field launch surface; ctl "start"), targeted owner-mode despawn (ctl named "stop" - one terminal command) and attach, the Appendix-B baseline self-mode stop, define-persona, purge, launch, and the resume/preservation family (resume-preserved / commit-resume / finalize-resume / prepare-preservation / commit-preservation / abort-preservation). One core, two thin doors (the panel's checklist-8 rule): the ctl dispatch branches for the resume family moved into door-shared methods; despawn/attach split into the shared authorizeNamed policy step (a denial surfaces as permission-denied on the ep door) plus a shared post-authorization effect core; spawn/persona/purge/launch/ models delegate to the existing op methods. Ordinary commands run the SHARED admitControl chokepoint; the resume family deliberately sits before it on BOTH doors (those ops must run while resumeRequired fences ordinary work) behind its own state fences. Tier semantics on the ep door are the ctl PRIVILEGED semantics (own-child despawn/attach, own-persona redefine): every spawn-capable agent cred already holds the Appendix-B owner-mode request rows, so serving them with admin reach would silently escalate those agents. Operator cross-agent reach stays on the ctl admin tier until the 1c grant-migration table mints admin-grade ep instruments. child and ledger authz modes are declared NOWHERE (the 1b gates: no fabricated ledger authority in static mode; child mode fails closed by absence until the durable spawner record lands). ps/inspect/spawn replies now carry each agent's lifecycleUid - the coordinate a targeted request pins; the serve table wires a fresh in-memory target resolver (static mode has no mapping-revision dimension; the revision is the constant 0). New manager-service-ops smoke (in smoke:ci), 39 checks over a real Manager + JWT broker + real stub agent processes: the rev-2 describe surface (17 commands, modes declared, child/ledger absent), ctl-vs-ep spawn parity proven field-for-field at the single startAgent chokepoint (identical StartAgentOpts + spawner attribution), real spawn -> ps/inspect -> targeted despawn with permission-denied (non-spawner), expired (stale uid) and not-found negatives, the spawned agent's own baseline cred self-stopping over ep.one, persona ownership through the ep door, the preservation fence (prepare -> ordinary ep ops unavailable -> abort restores), resume negatives answering with the exact ctl core refusals, and a sanitization section: traversal spawn ref / identity override / persona name / launch runId all refuse at the shared token grammar before any file effect, no stray file escapes the agents dir, and a newline-bearing model field cannot inject frontmatter policy (the YAML library owns quoting). Full typecheck + smoke:ci green.
…pc store (P2 item 1, 1c.1) Callers can now fetch-verify-compile every digest the manager registers, closing the in-memory-content-store residual the 1a panel named and laying the read path the generic describe/invoke CLI (item 5) rides: - core gains ensureContractStore (the EPC stream's create-or-verify, extracted from createEndpointStreams with the ensureAuthorityStores discipline: 13.12 flags verified on every boot, drift fails loud). The static manager ensures it at start beside the authority stores; the provisioner profile gains the EPC stream's CREATE/INFO grants. - the endpoint-serve-executor profile gains the space's epc publish subject: registration publishes every DISTINCT command schema root, its single-member closure manifest, and the cluster document + ITS manifest, all create-only content-addressed (a republish is an idempotent lost-CAS) and all BEFORE the spec registration that advertises their digests. Named residual: the one-shot executor can publish arbitrary new digest-addressed artifacts (unreferenced artifacts carry no authority; verify-on-read makes a wrong-subject write unservable). - every agent credential's Appendix-B BASELINE now carries EXACTLY ONE epc-subject-scoped Direct Get row (the last_by_subj form the client uses; never the bare/stream-wide form): describe answers digests, and a caller that may describe may fetch the schemas those digests name (content-addressed public artifacts; the per-caller authorization surface is the describe view, never the schema bytes). The D32 matrix audit exempts exactly that row shape from the untrusted-profile control-surface prohibition, with the safety argument recorded in the audit (public artifacts, create-only + deny_delete/deny_purge, verify-on-read; every other read verb and every non-EPC stream stay prohibited). - the manager contract module derives the compiled pairs, the cluster document, AND the published artifacts from ONE rows table (source schemas), so the served validator, the registered digest, and the fetchable artifact cannot drift apart. The ops smoke grows to 43 checks: fetch the cluster document at its REGISTERED closure digest over a plain agent cred, fetch a command's input-schema closure and RECOMPILE it to the registered digest (the generic-invoke round-trip), assert the recompiled validator enforces the same closed contract - plus the panel's 1b coverage NIT: a spawn-cap-only cred is broker-refused (no publish grant, no reply) on an admin-class ep command. preserve-state gains its missing npm script and joins smoke:ci (the tester's finding; it is the ctl-door integrity proof for the extracted resume branches). Full typecheck + smoke:ci green.
… describe/invoke caller (P2 item 1, 1c.2a) The panel live-confirmed a permanent shadow-DoS in the §13.7 contract store: the `epc.*` publish grant is a raw JS publish and the create-only fence is a publisher-SET header the grant cannot compel, so a non-cooperative grant-holder could APPEND a second message to a published digest subject; `last_by_subj` then returned the shadow and the fail-closed read made the honest artifact permanently unfetchable (deny_delete/deny_purge block recovery). Shadowing a cluster-document or manifest digest would break describe/invoke for every command - a total, operator-reprovision-only outage once ep is the sole contract path. Two independent remedies (belt-and-suspenders, both live-verified): - BROKER-ENFORCED per-subject immutability: ensureContractStore sets max_msgs_per_subject:1 + discard:new + discard_new_per_subject:true on the EPC stream (create AND an idempotent update for a pre-hardening stream), so a second publish to an occupied digest subject is broker-rejected (err 10077) regardless of headers. Boot-verify checks ALL THREE flags - a drift back to the default discard:old (which would DELETE the honest artifact, the footgun the critic's config matrix caught) fails loud. The provisioner gains the EPC stream CREATE/INFO. - VERSION-AGNOSTIC read fallback: fetchContractArtifact verifies last_by_subj and, on a verify-miss, falls back to the create-only WINNER (first-by-subject) and verifies that - the honest artifact is always first, every shadow is a strictly later append, so the reader recovers it on any broker/stream that lacked the per-subject cap. Only when neither the last nor the first verifies is the store genuinely corrupt (fail loud). publishContractArtifact's CAS-loss catch also treats the config-B reject (10077) as an idempotent loss. The upgrade path HEALS OR FAILS, never cements: applying the per-subject cap TRIMS each subject to its newest message, so upgrading an already-shadowed legacy store would keep the shadow and DELETE the honest create-only winner. ensureContractStore refuses the upgrade loud when a pre-hardening store cannot be proven clean (some subject holds more than one message, or the broker does not report num_subjects for a non-empty store), leaving the store untouched - still readable via the fallback - and directing reprovision instead of silently trim-cementing a pre-existing shadow. The false "never corrupt or replace" residual (endpoint-binding.ts + the provision.ts executor comment) is corrected to the accurate broker-enforced-immutability statement. Upgrade-path regression (tester, critical): the endpoint-serve-executor profile was missing the EPC Direct Get READ grant that publishContractArtifact's lost-CAS verify needs, so the manager exited on its SECOND boot (every re-up re-publishes the artifacts and loses the create-only CAS). The executor now holds both Direct Get forms (subject-scoped last_by_subj + bare next_by_subj fallback). Also lands item 5's substrate: endpoint-invoke.ts - a generic caller that describes an endpoint, fetches its registered contracts from the store, recompiles the digest-matching validators, and invokes by name, so a control consumer no longer hand-imports the endpoint's schemas. describe rides a purpose-built raw request (it pins no digests, so it can't ride the digest-stamping epCall). Smokes: ep-contract-store +17 (shadow-defense: config-B reject + still-fetchable; legacy-stream read-fallback recovery + both-garbage loud; upgrade-path harden-in-place with survival; config-A delete footgun corrected to B; shadowed-legacy upgrade refuses loud without trimming) = 57; new manager-service-invoke (describe→fetch→recompile→ invoke against the real manager WITHOUT importing its contracts) = 10; manager-service +3 (the re-up regression) = 25. Full typecheck + smoke:ci green (exit-verified).
…onto the ep rails behind minted instruments (P2 item 1, 1c.2b) The v0.3 ctl consumers now ride the v0.4 endpoint rails on static-auth meshes, through the GENERIC invoke path (describe -> fetch the registered schemas from the SPEC 13.7 store -> recompile digest-verified validators -> invoke) - no consumer hand-imports the manager's contract module. The ctl tiers stay dual-served; 1d deletes them once the remaining consumers (below) cut over. TIER LIVES IN THE CALLER'S GRANT (the ratified 1c grant-migration table): - operatorInstrumentCapabilities(tier) in core: the privileged instrument set is reads (status/ps/inspect/models) + spawn + define-persona, nothing targeted; the admin set adds ANY-mode despawn/attach (tOwner "*", operator-policy-mintable only per SPEC 13.2) and the untargeted manager.admin family (purge/launch/resume+preservation). - The control-caller-privileged / control-caller-admin / deployer profiles mint their tier's rows (plus the wildcard describe form and the one subject-scoped epc Direct Get fetch row); the user-mode deployer view (privileged) additionally holds an untargeted launch row whose authorization stays owner-equality. Instrument mints now REQUIRE a lifecycleUid (the caller triple pins the reply rail; fail loud). - The agent profile's `admin` capability arm mirrors the full admin instrument set (exact v0.3 parity: that capability granted the whole ctl admin tier); the `spawn` arm gains define-persona + inspect (the connector's persona write and per-agent read). Everything else about the agent surface is unchanged - agents hold no any-mode rows, and the ops smoke proves an agent cred publishing the any-mode form is broker-dropped. THE 1c ADMIN-REACH DECISION (rev-3 cluster document): operator cross-agent despawn/attach ride authz-mode `any` on the SAME commands (no wire-synonym command). The manager declares modes ["owner","any"] (revision 3) and maps a broker-authenticated any-mode request to its admin authorization path; on a user mesh the caller's CURRENT ledger scope must still carry `admin` (epAdminReach - the same fresh-read authority psOwnerFilter consults; an any-mode caller whose grant lost admin refuses loud, never silently downgrades to the owner path). `launch` serves with the ledger-derived admin flag, so the user-mode deployer keeps owner-equality while static/admin callers keep operator semantics - capability-only, exactly the 1b pin. DESCRIBE-BOUND INVOKE CURRENCY: describeEndpoint now returns the answering incarnation's instanceId+epoch off the reply SUBJECT (which the serve publish grant pins - a responder cannot stamp another's), and invokeCommand's default currency accepts exactly that incarnation, refusing failed-precondition when a different instance wins the `one` queue (a superseded-or-split manager; re-resolve adopts a legitimate successor). Strictly stronger than the ctl rail (which had no currency check at all) and needs no registry read grant on the caller. CONSUMERS MIGRATED: - CLI: askManager's static path connects raw, resolves the manager generically, maps the v0.3 op names to the typed commands (start -> spawn, named stop -> despawn, status -> inspect, camelCase admin family -> kebab wire names), resolves targeted aliases to (owner, actor, lifecycleUid) via ps, and rides mode any on the admin tier / owner on privileged. connectOrExit mints instruments with a fresh lifecycleUid and returns the caller triple; up's resume/preservation instrument does the same. - New `cotal describe <endpoint>` and `cotal invoke <endpoint> <command> [--args json] [--name <agent> | --self] [--admin]` expose the generic surface (docs/cli.md + the connector docs bundle updated). - connector-core: CotalEndpoint gains invokeService (per-endpoint resolve cache, one re-resolve on a currency refusal) and a public `principal`; the mesh agent's spawn/despawn/self-stop/purge/ define-persona ride it, resolving named despawns through inspect. Undefined args are stripped before the closed input contracts validate. - contractStoreContext constructs its JetStreamManager with checkAPI:false so fetch-only instrument creds need no $JS.API.INFO. - The auth callout's elevated-view branch passes the bearer's (already asserted, row-current) ledger lifecycle claim on the mint principal, so a user-mode instrument view (deployer / control-caller-*) mints its lifecycle-keyed ep rows instead of refusing. STILL ON ctl (enumerated, all pre-1d): open meshes (no service registry - the manager does not register there yet), user-mode bearers (the callout-minted caller triple is the named 1c.2c follow-up; the connector and askManager branch loud, and user-mode spawn parity is smoke-pinned), raw --creds files from older generations, the console's in-session ps/stop, and manifest live's deployer calls. Smokes: ep-grants 42 (instrument rollups + the any-mode grant row shape + the widened spawn set); manager-service-ops 48 (NEW: the operator instrument resolves generically with the describe-bound default currency and tears down a foreign-spawned agent via any-mode despawn; an agent cred's any-mode publish is broker-dropped); spawn-args rewritten for the invokeService door (+ the user-mode ctl-branch guard); ep-binding snapshot count updated; manager-split (130) and control-reply-bound (8) mint their instruments with lifecycle uids; flag-inventory + docs bundle regenerated. Full typecheck + smoke:ci green (exit code echoed and read from the log). user-spawn:live's despawn-retirement red is PRE-EXISTING (reproduced identically at the parent commit in a detached worktree; ungated suite, tracked in the plan).
…mpty form (1c.2b tester blocker) The tester's user-flow verification at 44208a2 caught `cotal stop --name <n>` unconditionally broken on the ep rails: the CLI strips the alias into the target block and has no args left; absent args ride the wire as null; despawn's all-optional `{type:"object"}` input rejects null, so every targeted stop refused `bad-request: / must be object`. The regression smokes missed it because they always passed `{graceful: false}` - never the no-args shape the real CLI produces. Fixed at the ONE generic layer so every consumer (CLI, connector, future callers) gets it right and no schema widens: invokeCommand marshals `undefined` args into the CONTRACT's canonical empty form - when the resolved input contract rejects null but accepts the empty object, send `{}` (that IS "no args" in that contract's vocabulary); a VOID input keeps null; an input that requires fields accepts neither form and still refuses loud at the pre-publish validation. Reproduced red-first: the ops smoke's SS11 despawn now invokes with NO args (the exact CLI shape) - it failed with the tester's bad-request before the fix, 48/48 after. Verified end-to-end at the user level: real mesh, real opencode agent, `cotal stop --name worker` succeeds, roster empty after. Full typecheck + smoke:ci green (exit code read from the log).
…to user-mode + close the cold-review HIGHs (P2 item 1, 1c.2c) Finishes 1c: every manager-tier control consumer now rides the v0.4 ep rails on BOTH static and per-user meshes, and the freelance cold review's two HIGH findings against the 1c.2b describe/launch paths are closed. USER-MODE END TO END: - The manager registers its v0.4 service on user meshes too (the registration/serve machinery is operator infrastructure over the space's static trust material, which a user mesh retains). The §13.7 contract-store ensure is hoisted out of the static-only lifecycle reconcile into a mode-neutral pre-registration step (a provisioner one-shot creates-or-verifies the EPC stream) - a live repro showed a user-mode manager registered against an absent stream and its artifact publish died no-responders. - The CLI's bearer path derives its caller triple (owner, actor, lifecycleUid) from the bearer's ledger claims; askManager's ep branch now accepts bearer+sentinel connects as well as static creds. - The connector's endpoint identity IS its caller triple in every auth mode (the endpoint's bearer-derived principal + the launcher's lifecycle uid), so the ctl branches are deleted from connector-core entirely; spawn/despawn/self-stop/purge/define-persona are uniform. - The auth callout's elevated-view branch passes the bearer's ledger lifecycle claim, so a user-mode instrument VIEW (deployer / control-caller-*) mints its lifecycle-keyed ep rows. - `cotal spawn -f`'s deploy probe drives ps/launch over invokeService for both the static admin credential and the user-mode deployer view. COLD-REVIEW HIGHS (freelance, ran the actual attacks): - HIGH #1 - describe was not request-bound: describeEndpoint subscribed the whole caller reply rail and accepted the first {ok:true} without matching the reply nonce or echoing the request id, so any live responder (the serve-publish grant spans every caller-nonce suffix) could inject a describe reply and capture the describe-derived invoke currency. Now request-bound exactly like epCall's parseAttributedReply: the reply SUBJECT's endpoint + nonce AND the body's echoed request id are checked; a non-matching reply is IGNORED (not rejected, so a racing wrong-nonce injection cannot fail an honest describe). New hermetic endpoint-invoke.smoke.ts reproduces the exact injection and proves it ignored + that a lone injected reply deadline-exceeds rather than being adopted (5/5, in smoke:ci). - HIGH #2 - deployer-view launch attenuation: the ep launch handler keyed its admin flag on epAdminReach, which reads the ACTOR's ledger scope and ignored the deployer VIEW's privileged-tier attenuation, so an admin user's (stolen) deployer bearer could bypass owner-equality (operator cross-owner launch) despite holding no admin rows. Since the deploy path is launch's only consumer and its spec stamps the caller's own owner (owner-equality always holds for a real deploy; static is single-owner so the flag is a no-op), launch is now uniform owner-equality (admin=false) on the ep door - the divergence removed in the least-privilege direction. Also folded from the review: - MEDIUM #3 - the down-manifest teardown (`cotal down -f`) was a ctl-admin consumer missing from the inventory; MIGRATED (not just enumerated): the teardown profile gains the admin instrument ep rows (lifecycle-keyed), and its ps/stop ride invokeService (ps + any-mode despawn resolved to each agent's current principal triple). - MEDIUM #4 + wording - define-persona admin=false and launch owner-equality are documented as TWO deliberate, panel-accepted least-privilege narrowings (not bit-exact ctl parity); the "any/admin rows only in operator instruments" comment now correctly includes the admin-capability agent credential (parity with its old ctl.admin authority, not a new escalation). - The `manager.admin` family (purge + the resume/preservation ops) now re-checks epAdminReach AT SERVE TIME on user meshes (the adminGated wrapper, security4's 1c.2b follow-up), so a revoked scope demotes the next call instead of riding the bearer's remaining row lifetime. - The static epAdminReach=true residual (a leaked static admin instrument retains reach until its TTL, the campaign's static-revoke class) is named in the doc (critic's non-blocking ask). CONSOLE reclassified to the 1d bucket (not migrated): its self-minted "admin" profile credential is read-only (no ctl publish), so its in-TUI ps/stop only ever functioned on open meshes / explicit --creds - both already 1d decisions. Smokes: endpoint-invoke +5 (the injection binding); user-spawn:live +3 (a user bearer resolves the manager generically + invokes inspect over ep + a spawn-scope bearer's ps is broker-dropped); spawn-args user-mode section flipped to the invokeService door; ep-grants unchanged (teardown rows are the existing admin rollup). The teardown profile's ep rows make a uid-less mint fail loud, so the four smokes minting "teardown" bare (manager-split, control-reply-bound, standing-renewal, backup-live) now pass a lifecycleUid, matching the production mint in down-manifest's mintIfAuth. launch-parity's client-window check (#159 B1) now fakes invokeService instead of the deleted requestControl door and also pins the invoked command name. Full typecheck + smoke:ci green (exit code read from the log). The pre-existing user-spawn:live despawn-retirement red (no responders on ctl.auth-admin) reproduces at 505d99a - out of scope, tracked separately.
… v0.4 service endpoint the only control door (P2 item 1, 1d) Core: drop the manager/self/admin control tiers, ControlTier, and controlSubject (the delivery/delivery-admin/auth-admin carve-outs stay); every credential profile is ep-only. Agent baselines lose the ctl.self publish and control-reply subscribe rows, the supervisor serves no control tier, and the operator instruments carry endpoint capability rows only, so the old manager control subjects are unreachable end to end (publish rows, serve subscriptions, and handlers all gone). Manager: the three-tier serveControl and the handle/handleActive routers are deleted; the service registers on every mesh. Auth meshes ride the scoped endpoint-serve executor; open meshes run the same gate/registration/serve-grant ceremony over bare one-shot connections (no credential is ever minted) and create-or-verify the authority stores at boot, closing a live-reproduced crash where a raw broker died "stream not found" at the first gate write. ps/inspect rows pin `role` optional: a manifest-launched agent declares none and the responder failed its own pinned output schema (live-reproduced). CLI: ControlTier becomes ControlReach (owner/any). askManagerEp resolves the target via inspect and derives the authorization mode from the resolved owner (an own-domain target rides owner mode; a cross-owner target rides any mode, which the broker admits only for admin-instrument holders); open meshes ride a bare caller triple; a raw --creds control caller without an ep caller identity refuses loud. The console drives stop/ps over invokeService. Smokes: the ctl matrix re-pointed at the ep rails (control-auth 13/0, manager-split-auth 131/0 with the ctl-unreachability proofs, manager-service* 80/0, self-serve-join-auth 23/0, channels-auth, e2e-acl 23/0, cross-owner-auth 24/0, views 42/0, preserve-state, self-serve-join-coverage 17/0, user-spawn, spawn-detach-live 26/0, readiness-window-live 11/0); ctl-trust re-pins the serveControl trust guards on the surviving ctl.delivery rail with a scoped delivery cred (6/0); control-reply-bound is deleted with its rail. user-spawn boots the W6 plane-liveness oracle for the auth-service restart (the SIGKILLed predecessor's plane claim otherwise refuses fail-safe forever; a pre-existing red reproduced at the 1c.2c base) and keeps the cli alias on one lifecycle across scope upserts (the R1 takeover barrier refuses issuance under a rotated grant while the predecessor is active); its ps assertions move to the v0.4 instrument-gated read model (a raw spawn bearer cannot enumerate at all), and its stale-bearer crossover section pins the R1 refusal explicitly with the positive half proven on a fresh alias. spawn-detach-live sandboxes the operator config dir (the real installed-extensions store leaked into the subprocess sandbox) and ext-adds the worktree connector for the foreground path. Full smoke:ci gate green (82 suites) plus the out-of-gate live matrix above at this tree.
…bind leaf) The manager will serve spawn as an inline-accept action (Model B): its handler binds the goal at accept and commits its terminal off-handler, so it needs a dedicated standing goal-writer connection distinct from its serve credential. goalWriterGrants composes commitPrincipalGrants with the goal .bind leaf the canonicalizer normally owns, so one principal covers the whole accepted->terminal goal-fact chain of its own endpoint; the serve credential holds none of these. Additive only (no existing grant profile touched); asserted in ep-binding smoke.
The RED-FIRST gate for 'spawn becomes an action': a real Manager + JWT broker
exercising M1 accept shape {goalId=env.id, fingerprint, executor}, M2 progress
order, M3 three terminals, M4 settle race, M5 kill/reconcile under the epoch
fence, M6 same-alias refuse, M7 serve-cred broker-denied a goal write (+ pins
2/3). Against today's blocking spawn M1-M6 fail by design (7 passed / 15 failed):
that red is the slice's first artifact. M7 + the pin-2/3 boundaries already pass
(the serve cred and callers never held goal writes; caller-triple containment is
structural), and stay green after the handler surgery. Wired into smoke:ci.
The manager's spawn-as-action handler binds + commits goals on a dedicated standing connection disjoint from its serve credential (Q2). This profile mints exactly goalWriterGrants(space, endpoint, connId) for that connection — the goal bind/terminal facts + goal-record writes + fencing reads, inbox-scoped — the endpoint-serve-executor least-privilege class the panel named. Typecheck green.
A standing connection + ActionContext for spawn-as-action, disjoint from the serve credential (Q2): auth mode mints the scoped goal-writer cred (goalWriterGrants only), an open mesh uses a bare connection. Stood up in start() after registration, drained on both stop paths. Runtime-verified: smoke:manager-service 23/0 with the goal-writer standing (auth). Not yet consumed — the accept/async handler is next.
… still numbers) A hard-pinned name (imperative --name/identity override or a manifest-declared name via opts.resolved) colliding with a live/provisioning/reserved incarnation now refuses loud at accept, before any reserve/mint/bind (pin 1) - never a silent -2 suffix, so an address-by-triple caller's pinned name can't be re-pointed. A persona-derived base name (no pin) keeps uniqueName's numbering (multi-peer spawn). Verified: typecheck 0, smoke:manager-service-ops 47/0.
The accept seam for spawn-as-action: onAccepted fires after the incarnation identity is minted but before any provision/side-effect (binds the goal + replies the acceptance; a throw aborts before provisioning, the finally releasing the reserve = the bind-conflict refusal, pin 1). onLaunched fires after runtime.spawn (the launched progress edge). Blocking callers (roster boot) pass none - unchanged. Typecheck 0.
…t a bucket scan My first fix for this used liveKvEntries, which binds a push consumer over the bucket. The supervisor's grant on the manager lease bucket is STREAM.INFO, STREAM.MSG.GET and the lease.* publish, and nothing else; the CONSUMER.CREATE grants next to it are on the presence bucket for the roster watch. So that version raised a permissions violation on every lease read on an authed mesh, for exactly the callers this probe serves. It replaced a false empty inside a five second window with a hard failure on every call. The suite could not see it. It runs an open mesh, which has no permissions to violate, so the fixture made the failure unrepresentable rather than merely untested. This reads STREAM.INFO with a subjects_filter to learn which instance keys exist, then one last_by_subj point-get per key, which is the shape the grant already allows and the one used for the quiescence check. The distinction that matters: last_by_subj is correct PER KEY and wrong ACROSS keys. Scoped to one instance's subject it returns that instance's latest state, so a tombstone retires only its own key. The defect was asking a single wildcard for the newest message in the whole subtree, where a stopping peer's tombstone outranks a live sibling's older put. A key that disappears between the info call and its get is skipped rather than raised: it is not a live holder, and that is an answer rather than a failed read.
… die on it The bare-subject read is already gone; this is the other half. STREAM.INFO lists TOMBSTONED subjects, so a cleanly released `lease.<instanceId>` is in the enumeration, and the point-get was called without `allowEmpty`, whose default throws on a marker instead of returning undefined for the holder check to skip. A cut against a space where one manager had been stopped cleanly died naming that dead manager's key, before any live holder later in the walk was examined. Both sibling walks in the same function already pass `true`, and the presence one carries a comment saying exactly why. This loop was copied from them without the argument. Two loops, one function, twenty lines apart, and nothing compared them. Trigger is a clean stop of a SECOND instance, not a restart: a restart reuses its instanceId, so the new put supersedes its own tombstone on one subject and the walk sees a single key. Two workspace roots, or a manager retired for good. Also aligns the filter to `lease.*`, matching the endpoint-side probe. `managerLeaseKey` is contractually one lowercase-alnum token, so `*` matches exactly the key space that can exist while `>` would admit a malformed multi-token key. Two probes of one bucket disagreeing on their wildcard is drift the next reader has to adjudicate from scratch. THE CELL, AND WHY ITS FIRST VERSION WAS WORTHLESS. A regression test here has to run through the shipped function, so `readPresenceWithoutConsumer` is exported: a transcription carries its own `allowEmpty` parameter and stays green when the real argument is removed. That was necessary and not sufficient. The first cell wrote the tombstone first and still passed with the fix reverted, because write order is not enumeration order: `Object.keys` follows what STREAM.INFO returns, which is sorted, and the live id sorted first, so the walk broke before the tombstone was ever fetched. Both cells silently tested nothing. The cell now picks ids whose sort order IS the intended visit order AND reads the enumeration back to assert it got that order. Choosing the names is the fix; verifying the premise is what stops it drifting back into testing nothing. Both orderings are asserted, so it cannot report a frequency it did not measure: in production instance ids are random lifecycle uids, making this a coin flip rather than a deterministic failure. Verified by mutation rather than by colour. With the argument removed the cell fails at the tombstone-first case naming the released subject, while its two premise checks still pass on the same broken build; restored, eight checks green. Not covered: the full `down --preserve-state` cut against a tombstoned sibling is still unexecuted by anyone. This reaches the function the three production callers use; it does not drive the command. Nothing in `smoke:ci` loaded this path before, so the suite is also the first gate coverage it has had.
… its fixture does not work Both security seats asked for this and they are right to: `manager-lease-probe` runs an open mesh, which has no permissions to violate, so a `readManagerLease` implemented as a whole-bucket scan passes every one of its cells and then raises an Authorization Violation on every call against a real mesh. That happened earlier today. The dimension is missing, not the coverage. The suite is here with the two cells that would close it: the shipped probe read under a real supervisor credential with a tombstone present, and a consumer bind REFUSED on the lease bucket while the SAME bind on presence is ALLOWED. The allowed arm is the point; a refusal alone proves nothing, because a broken probe also fails. It does not run. The fixture raises an Authorization Violation during setup, before its first cell, so the failure is in how this file stands up auth and not in anything it is trying to measure. It is therefore declared but kept OUT of the smoke:ci chain, with the reason recorded in the gate inventory rather than left as an unexplained absence: an unrun suite with a recorded reason is a decision, without one it is debt. Shipping it gated would have put a red in the chain for every lane. Deleting it would throw away the only cells anyone has proposed for the blindness that let a grant-violating read pass a green suite. Neither is right, so it sits in the middle with its state written down.
The open-mesh lease probe cannot represent a permissions refusal, so a readManagerLease implemented as a whole-bucket scan passes every one of its cells and then raises an Authorization Violation on every call against a real mesh: the supervisor's grant on that bucket is STREAM.INFO + STREAM.MSG.GET + the lease.* publish, with no consumer verb. This suite is that dimension. Two cells, and the second is the point. Cell A reads the live holder under a real supervisor credential with a cleanly stopped sibling's tombstone present. Cell B binds a push consumer on the manager bucket and on the presence bucket under the same credential in the same run: presence is allowed, the manager bucket is refused. The allowed arm is what makes the refusal mean something, since a broken probe also fails. The fixture was previously declared ungated because it raised an Authorization Violation in setup. The cause was not a grant fact. setupSpaceStreams was called positionally against a signature that takes an options object, so opts.servers was undefined, the client fell back to its default target, and the fixture was running against whatever broker was on the default port while holding no credential for it. Being pointed somewhere else looks exactly like being refused, so it now asserts its own target before it reports anything. No smoke file is in the typechecked tsconfigs, so nothing but running it could have caught the arity. Second fix in the same file: the seed and the consumer-bind probe connected with a bare credsAuthenticator, leaving the default _INBOX. prefix while a scoped grant allows only _INBOX_<connId>.>. A push consumer's deliver subject is an inbox subject, so the ALLOWED arm would have been refused on its subscription and cell B would have reported no discrimination for a reason unrelated to the grant under test. Both sites now go through standaloneConnectOpts, which derives the prefix from the credential. Gating it removes its gate-inventory entry: the recorded reason described a fixture that could not reach its own broker, and a reason that outlives its state documents something untrue. The chain goes from 169 to 170, which reshuffles the CI shards positionally, so a red on the next roll can be a suite that changed runners rather than a regression.
… reply `spawn -f` recorded each launched agent by copying `requested` and `hash` out of the manager's launch reply through an unchecked cast. P2 item 2 ruling 3 had already taken those fields off that reply on purpose: the launch output contract is the action acceptance floor (allocated identity plus goal coordinates), and the manifest details are documented there as re-derivable by the deploy caller, because the caller submitted the runId and the name. The writer kept reading them from the wire, so both arrived undefined, JSON.stringify dropped the keys, and every deploy that launched an agent wrote a row missing two required fields. Nothing failed at write time. It failed at teardown, where loadLedger refuses the row, listLedgers skipped the refusal silently, and findLedgerByHash reported "no ledger matches this manifest's current contents (was it edited since spawn -f?)" -- an accusation about a file the operator never touched. The record was unreadable from the moment it was written; no crash or restart was involved in producing it. The fix takes both fields from the plan entry that is already in scope at the write site (e.agent.name is the manifest key, e.hash the resolved drift hash) and takes only name, id and lifecycleUid from the reply, which is exactly what the acceptance floor carries. The manager and core are untouched: putting the fields back on the wire would reverse a deliberate narrowing inside a bug fix. Exactly one production site read those fields off a launch reply, and it is this one. Every other consumer of `requested`/`hash` reads the ledger, not the reply: spawn-plan.ts keys drift detection on prior rows, down-manifest.ts derives cred paths. So the defect had one site and this closes it. Also hardened, because the cause survived three chances to be caught: - The row is now parsed against the schema that will read it, and a field the reply stops carrying fails loudly at write time naming the field, instead of persisting a record the reader refuses. - listLedgers returns refused files alongside valid ones, and findLedgerByHash reports them. A file named for its own runId is this root's record, so a validation failure on it is our bad record rather than a foreign file. The validator's message already named both missing fields; it was being discarded by a bare catch. smoke:ledger is gated (chain 169 -> 171 with the authed lease cell) and gains a round trip: a row the writer produces, written and read back through the reader `down -f` calls, plus the pre-split no-uid case and an arm proving a reply that loses the spawned id is refused rather than persisted. The reply it is driven from is the acceptance floor, not a literal composed to be invalid, so it asks whether the writer produces a readable row given what the wire actually carries. Reverting the field sourcing aborts that cell naming both fields. Adding an entry reshuffles the CI shards positionally, so a red on the next roll can be a suite that changed runners rather than a regression.
|
|
||
| try { | ||
| let up = false; | ||
| for (let i = 0; i < 100 && !up; i++) { |
B6: `--on` cannot work on an authed mesh. epRequestGrantRows emits the instance form ep.inst.<endpoint>.<instanceId>.<command> only when a capability carries instanceId, and no profile sets it -- operatorInstrumentCapabilities builds one/all routes and nothing else. The code already names the gap at its own `ps` route decision: "the other reads stay one-only (anycast, or inst when a resolve pins --on)". The route is reserved in the grammar and never minted. The probe is two-armed because issuing the capability makes the allowed arm pass trivially, and a mint that hands the instance route to everybody passes it just as well while being a security regression. Both directions are measured in one run from the shipped builder: operator instruments should reach an instance route once the capability exists, and ordinary principals must never hold one, before or after. An agent carrying the spawn capability is not an operator. Today the denied arm is already green and the allowed arm is the red, so the denied arm is written first and has to survive the fix rather than being added after it, when its shape would be chosen by whatever the mint happened to do. The control cell earned itself immediately. It asserts every profile under test reaches an ordinary route first, and that caught three fixtures that would have produced a vacuous denied arm: operator, admin and deployer hold no ep request rows at all, so "no instance row" is true of them for a reason that has nothing to do with instance addressing. The ep-caller profiles are control-caller-privileged and control-caller-admin. An observer holds no request rows either; it is recorded rather than asserted, so the sweep stays honest about who is in it. Ungated with a recorded reason: it reproduces an open defect, and gating a known red teaches every lane to skim reds. Gate it when the mint lands. Scope: this reads the rows a credential is MINTED with. A row is what the broker enforces on, but a minted row is not an observed refusal, so the enforcement arm -- publishing an instance-route request under each credential and reading the broker's verdict -- is owed separately and is not in here.
…against measurement Both security seats required `deployer` on the allowed arm: it is the `spawn -f` / `--on` path, and omitting it was the vacuity trap in the other direction, an allowed arm missing the one profile the flag actually runs under. It is now measured, and so is the reason it was missing -- called without a lifecycle uid it reports zero ep rows, so my earlier reading of "deployer holds none" was an artefact of the call, not a fact about the profile. With the uid it holds 18. The control now checks a DECLARATION against the measurement rather than asserting a floor. Each subject states whether it is an ep caller and cell 1 fails if reality disagrees, in either direction. That closes the same defect in a third position: a profile holding zero ordinary ep rows sits in an arm proving nothing and does so silently, which is true of operator, admin, and a uid-less deployer. A profile that later gains or loses ep rows now fails loudly here instead of quietly weakening whichever arm it sits in. The denied set is unchanged and keeps the spawn-capable agent alongside the plain one, since the capability builder claims exactly that boundary. The observer is declared not-an-ep-caller rather than special-cased by name, so its zero is an assertion instead of an exemption. Shape follows the panel's answer to the re-ask: B6 is C, the exact mint-time instance id on the per-invocation instrument, not a standing wildcard row. The allowed arm is red for all three subjects until that mint lands.
`--on <instanceId>` could not work on an authed mesh. The instance form ep.inst.<endpoint>.<iid>.<command> is emitted only when a capability carries instanceId, and no operator instrument ever set one, so every instance-targeted route was refused at the broker while the grammar reserved the subject. The reason this is the right shape, in the order the evidence supports: Established: the control-caller instruments are one-shot, five minutes, minted per control call, and the resolve that pins the instance happens before the mint. The exact id is therefore in hand when the credential is built, so the least-privilege issuance is one exact row for this invocation and nothing else. No standing wildcard is needed and none is added. Supporting, and measured rather than assumed: reaching a chosen instance with only the anycast route means retrying until you land on it, which dispatches the command to other instances on the way. On a three-manager fleet that was observed as state changes on instances the caller never addressed, roughly N-1 per arrival for a fleet of N. Cite N-1 rather than any single run's rate. That measurement was taken on a non-mutating command and on an open broker, so it establishes dispatch, not enforcement, and it is supporting evidence for why a pin must exist rather than the reason for this shape. Not claimed: a standing ep.inst.<endpoint>.*.<command> row. That is operator multi-manager addressability, it needs its own bar and an explicit product decision, and it is not smuggled in here. Nor is anycast-and-retry endorsed as a supported pattern by measuring it. The change threads the already-declared MintOpts.endpointCapabilities into the operator-instrument paths, which honoured no per-mint capability before. It rides the same `extra` seam the deployer's owner-equality launch row already uses, and the pin appends to a profile's standing set rather than replacing it. The emitter validates the token, so a malformed id fails at mint instead of widening a subject, and a wildcard cannot be passed as a value at all. The gated probe covers both directions. Ordinary principals hold no instance row, including a spawn-capable agent, since the capability builder claims exactly that boundary. The three operator profiles each hold exactly one, and the row is asserted to name the resolved id with no wildcard, because a cell counting rows alone would pass for the shape that was refused. One boundary is recorded rather than asserted away: the row builder is principal-blind. An agent handed a pinned capability does receive an instance row, so operator-only is enforced by the authority that chooses capabilities, not by the emitter. That is pre-existing behaviour of the agent path and is unchanged here; it is measured so the next reader does not assume the builder enforces a rule it does not. Still owed before ship: the runtime publish arm, since a minted row is what we issued and only a broker verdict is what is enforced.
Its ungated reason was that it reproduced an open defect, and gating a known red teaches every lane to skim reds. The mint has landed and all three operator profiles hold their exact pinned row, so the reason is spent and the entry is removed rather than reworded. Chain 171 -> 172, which reshuffles the CI shards positionally, so a red on the next roll can be a suite that changed runners.
…ranteed Both security seats ruled the principal-blind emitter a non-blocking residual and refused a builder inversion under a bar written about standing wildcards. The ship gate asked for an audit that no control path mints an agent with a pin, which is true today: the agent-mint sites pass explicit field-by-field opts and none lists endpointCapabilities. Written with its date and tip because true-today is a fact with a shelf life. provisionAgent spreads opts straight into the agent mint, so the invariant is held by callers choosing to enumerate their fields rather than by the grammar, and a future caller building opts from a payload would gain the capability silently. The label is unreachable from today's callers, unenforced by construction, which is weaker than a hole and weaker than a guarantee. Also recorded: the static managed spawn refuses endpointCapabilities outright with a stated reason while the dynamic path only omits it. The same door is bolted on one side, and asking why is a smaller question than inverting the builder.
… unpinned
The mint suite proves what we ISSUE. This proves what is ENFORCED, and they are
different claims: a minted row describes a credential, a broker verdict
describes the wire. The mint suite would stay green if every row we issue were
ignored, which is why both seats asked for this before calling B6 closed.
The discriminator is one mint input rather than two profiles. The same profile
is minted twice in the same run against the same broker, once with the resolved
instanceId and once without, and the instance route flips from refused to
accepted. Two different profiles would confound the pin with everything else
that differs between them; one profile against itself isolates what C changed.
PINNED control ep.one ALLOWED, instance route ACCEPTED
UNPINNED control ep.one ALLOWED, instance route REFUSED
The shape and the classifier are taken from a probe cs-lane-session executed and
then published a defect in, and both halves are load-bearing. A real row out of
permissionsFor is published as the control with only its trailing nonce made
concrete, then only the route segment is rewritten. Outcomes are three, never
two: no-responders means the broker accepted the publish and nothing serves it,
which is the positive signal rather than the absence of a negative one; a
permission error is a refusal; a timeout is void and is counted as neither arm.
Folding a timeout into allowed would let the control pass under load while
measuring nothing.
Its connect is deliberately not taken. It hand-rolls the inbox prefix from an
identity that equals the credential-derived one only by a coincidence of the
current cred shape, which its own author retracted. A drifted prefix gets the
allowed arm refused on its subscription, poisoning the control rather than the
claim and reading as a green denied arm. standaloneConnectOpts derives it from
the credential. The fixture also asserts it reached its own broker before
reporting, because being pointed elsewhere is observationally identical to being
refused.
B6 moves from mint-closed to enforce-closed. Chain 172 to 173, which reshuffles
the CI shards positionally.
Replaces freezeExpectedSet's kv.keys() ordered-consumer enumeration with STREAM.INFO + subjects_filter. Same subjects, same return, one read-only metadata verb instead of three consumer-lifecycle verbs. NOT DONE: the existing fixtures intercept kv.keys, which this deletes, so they no longer aim at the path under test. endpoint-serve's deadline cell now reports 'no throw' (it stalls a call that is gone). Every fixture across endpoint-serve + endpoint-virtual must be re-pointed and each one mutation-proved by disarming it, per the enumeration owed. Includes the arm-2 repro probe (user-mode ps, no crash, no replacement supervisor) that reproduced the regression.
The cell stalled kv.keys, which the consumer-free enumeration no longer calls, so it reported 'no throw' — a fixture aimed at a deleted call site. Stalls streams.info instead, and only that: the per-slot leader reads use other jsm methods and stay live, so this proves a stalled ENUMERATION rather than a stalled everything. Mutation-proved both ways: armed 133/0; with info passed through to the real jsm the cell reports 'no throw' and the suite exits 1.
…rom empty
The old cell stubbed kv.keys, which the consumer-free enumeration no longer
calls. Measured: with that stub removed entirely the cell still passed
(133/0, cell silent) — green on an empty {} jsm throwing a TypeError rather
than on a permissions failure.
Re-pointed at streams.info, and asserting the MESSAGE rather than only the
code, because the code cannot discriminate: a refused read and an empty
registry are both failed-precondition. A third cell asserts the two messages
actually differ, since a marker present on both sides cannot discriminate.
c() prints nothing on a pass, so a cell's name appears iff it FAILED. An empty grep is the success signal here, which is backwards from every other instrument and half the misreadings are the reassuring direction. Also says plainly that the suite exit code is not evidence about any single cell.
…sumer verbs The svc.<e>.*.spec enumeration is now a STREAM.INFO with a subjects_filter, so the scatter-freeze read no longer needs CONSUMER.CREATE/INFO/DELETE on the records bucket — three consumer-lifecycle verbs to list keys, replaced by one read-only metadata verb. Measured, not assumed: with the enumeration converted and this row absent, the static/operator cotal ps is refused on STREAM.INFO.KV_<records> — a path that works today. The conversion and this row land together or the operator path regresses.
Asserts the static/operator cotal ps works. It caught a real regression in its first outing: converting the freeze enumeration without the matching STREAM.INFO grant breaks operator ps, and the symptom surfaces hundreds of lines away as a NATS permissions violation. Appended to smoke:ci rather than grouped, so the new entry takes the last index and shifts no existing one — a positional shard assignment keeps every current suite on its runner.
It began as the control arm of a BEFORE/AFTER pair, where a static failure meant 'the fixture never armed, grade the pair void'. As a gated suite that reading is inverted and would train a reader to dismiss a real regression as a fixture artefact. It now says a red is a product defect, and the cotal up cell is checked first so the two are distinguishable.
The scatter freeze needs STREAM.INFO on the records bucket, which only the static control-caller-privileged instrument holds. A user-mode bearer never has that row, so scatter died on a permissions violation that read as "no manager" even when ep.one.manager.ps would have been served. Mode is chosen up front from the connection shape — bearer means ep.one, otherwise scatter — never try-scatter-catch-degrade. connectOrExit refuses control-caller-* on a user mesh (static-only instruments); resolveControlTarget translates to the user bearer path explicitly. Docs state the user-mode completeness bound. Gated by ps-operator-path (dead-manager honesty) and ps-user-mode.
freezeExpectedSet enumerates via STREAM.INFO and reads slots via STREAM.MSG.GET — both through jsm. The KV handle was a leftover from the kv.keys() path and nothing read it. epScatterService and scatterCommand stop opening a records bucket solely to pass it through.
The spawn-scope ps refusal is asserted in user-spawn.smoke.ts B1e, which is not in smoke:ci. Record that in the shipping artifact so it is a known limitation rather than a silent one.
resolveControlTarget routes user meshes through connectUserControlOrExit, which has no Profile argument — a placeholder "agent" role would be meaningless today and wrong if the user path ever consulted it. The bare catch around resolveTargetOrExit is gone: that helper exits on target errors and any other throw propagates. Double resolve (mode peek, then connect) is documented as an accepted cost of this slice.
| * Run: pnpm smoke:down-manifest-usermode:live | ||
| */ | ||
| import { spawn } from "node:child_process"; | ||
| import { execSync } from "node:child_process"; |
| import { execSync } from "node:child_process"; | ||
| import { createServer } from "node:http"; | ||
| import type { AddressInfo } from "node:net"; | ||
| import { existsSync, mkdtempSync, readFileSync, readdirSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; |
| if (cond) { pass++; console.log(` ✓ ${name}`); } | ||
| else { fail++; console.log(` ✗ FAIL: ${name}`, extra ?? ""); } | ||
| }; | ||
| const wait = (ms: number) => new Promise((r) => setTimeout(r, ms)); |
Only package.json conflicted; the rest of the tree auto-merged. Resolution: - take main's new `smoke:claude-wake` script and append it to the end of the `smoke:ci` chain. Appending leaves every existing chain index untouched, so no existing smoke changes CI shard. - keep `smoke:control-reply-bound` deleted. This branch removed it along with the manager ctl rail it tested; its smoke file no longer exists, so restoring the script would point the chain at a missing file. Chain is 176 entries: every entry resolves to a defined script and every script resolves to a file that exists.
…ward git auto-merged the generated bundle and the result no longer matched what the generator emits, so `check:docsbundle` failed. Regenerated from source.
…id-launch `awaitReadiness` clears its timeout at the top of the exit handler, then on the deliberate-stop path returned without resolving. That removed the only other resolver, so the promise stayed pending forever and `startAgent` never released its lifecycle ticket. Every drain that waits on that counter then blocks permanently: `preparePreservation` never replies, and a preserving `down` hangs behind it. It surfaces only when a stop beats the agent's first presence card. On macOS presence usually wins, so the suite passed; on Linux the stop lands first and `manager-service-ops` times out on `manager.prepare-preservation`. Settle it as its own variant rather than reusing `failed` or `uncertain`: the despawn path still owns the goal terminal and commits `cancel`, so the caller returns before the failed/uncertain arms and emits no competing outcome. Verified on Linux at the same tip: before, the suite hangs and exits 1 after 35s; after, it passes 52/52 in 11s.
…nch stop Settling readiness on the deliberate-stop path made `startAgent` return a non-ok reply, and the post-accept fallback in serveSpawnGoal turns any non-ok reply into a `failed` terminal when no terminal was entered. That raced the despawn's `cancel` and, when it won, reported an operator-cancelled agent as having died on launch (`manager-spawn-action` M4). The never-settling promise this branch just fixed had been masking that: by never returning, it also never reached the fallback. Add `onTerminalDeferred`, which claims the terminal without committing one, and call it on the deliberate path so the fallback stays quiet and the despawn handler remains the single writer of this goal's outcome. Verified on Linux and macOS: manager-spawn-action (incl. M3 process-exit-failed and M4 despawn-cancels), manager-service-ops, manager-spawn-action-auth and preserve-state all pass on both.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves the manager's control plane onto the v0.4 endpoint rails and deletes the private
ctlrail, then builds three capabilities on top of it.The manager becomes an ordinary endpoint
The manager registers as a
serviceendpoint on every mesh and serves its full typed command surface over the standard rails. Core gains the endpoint-serve credential subsystem: the issuance gate, the serve ledger, the registration barrier with fail-closed eviction, and a key-pinned one-shot executor profile scoped to a single endpoint instance.Every control consumer now rides those rails. The CLI, the connectors, and the deploy probe all go through describe, fetch the registered schemas from the content-addressed contract store, recompile digest-verified validators, then invoke.
cotal describeandcotal invokeexpose the same generic surface to operators. Invoke currency is describe-bound, so a superseded or split-brain manager refuses rather than answering stale.The migration then completes: the legacy
ctlrail is deleted. Core drops themanager/self/admincontrol tiers, theControlTiertype, andcontrolSubject. Agent baselines lose theirctl.selfrows, and the CLI replacesControlTierwithControlReach(owner/any) derived from the resolved target owner.Spawn and launch become actions
Asking the manager for an agent no longer blocks the caller. The manager accepts a spawn goal, returns the allocated identity immediately, and emits progress to a terminal outcome: presence within the readiness window settles
succeeded, an early exitfailed, and the window elapsing with neither isuncertain, which a laterpssettles against the live roster. A hard-pinned name colliding with a live agent refuses at accept, before anything is minted. A restarted manager settles any goal a predecessor accepted but never terminalized.One space can run several managers
Each manager persists a stable logical instance id across restarts and advances its process epoch on return, so peers address a specific manager regardless of which process serves it.
spawn --on <instance>pins one; an untargeted spawn rides class anycast;psandstatusscatter across every registered instance and label a non-answering one unreachable rather than omitting it. The manager lease is demoted from a per-space singleton to per-instance liveness.Attach rides a mesh session
cotal attachno longer returns a127.0.0.1websocket URL. The reply carries a signed, one-use, holder-bound session grant; terminal bytes then stream on session subjects scoped to the two parties, with backpressure surfaced as an explicit drop notice. A late attach repaints from a replayed snapshot, and close, expiry, target despawn, and manager restart are distinct end states. The browser console becomes a real mesh session client.Contract schema profile
Contract schema registration is bounded structurally. An unrecognised keyword is refused rather than ignored, because a profile that bounds a document cannot soundly bound what it does not recognise. The node-count ceilings are removed: neither candidate basis for the constant survived measurement. The compile and validate time budgets are reported rather than enforced, since no instrument on the supported Node floor measures the intended quantity, and enforcing them refused valid arguments and the manager's own contract at startup.
Breaking
compileContractis exported from the released@cotal-ai/core, so a contract schema that registered under an earlier version can now be refused. Check registered schemas against the admitted vocabulary before upgrading.ctlcontrol subjects are unreachable end to end. Publish rows, serve subscriptions, and handlers are all gone.CONTROL_PRIVILEGEDno longer exists in core. A connector built against an earlier core fails to import against this one. Extensions must be reinstalled when the binary is upgraded, and the failure surfaces as a loud import error naming the fix rather than as a silent degradation.Verification
Full gate green at
fb0510d6, 164 suites, recorded against an unchanged tree at both ends. The live user-path suites were also run directly:dogfood,up-stack,setup-pure,readiness,spawn-detach,user-spawn,up-manifest,manager-singleton,ext, andserver-resolutionall pass.Known reds outside the gate, all reproduced and none of them silent:
user-auth-launch:liveanddown-manifest-usermode:livefail on theCONTROL_PRIVILEGEDupgrade path described above, andweb-seed:livefails on a publish permission for its own ops subject.renewal-terminal-racereproduces a credential-renewal race against the durable journal and is deliberately outside the gate as an expected red until a fix lands.