Skip to content

Expose deterministic IDs and tighten TUI controls#111

Open
tony wants to merge 40 commits into
masterfrom
issue-80-deterministic-identities
Open

Expose deterministic IDs and tighten TUI controls#111
tony wants to merge 40 commits into
masterfrom
issue-80-deterministic-identities

Conversation

@tony

@tony tony commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Summary

Add deterministic, privacy-safe identities for normalized search records and
observed conversations, expose them consistently through CLI text, JSON/NDJSON,
MCP, and the HUD TUI, and disclose the accompanying pi-style TUI command pass.

This establishes three distinct versioned handles:

  • agc1: identifies semantic content from record kind, case-folded role, and
    exact normalized text.
  • agr1: identifies one defensible native or source-order occurrence.
  • agt1: identifies one defensible backend-native thread anchor.

Each handle uses the first 128 bits of a domain-separated SHA-256 envelope,
encoded as full lowercase base32hex. Unknown record or thread identity remains
null; agentgrep does not invent identity from a path, timestamp, or file mtime.

Design

  • Add a dependency-light identity module and adapter-owned topology metadata for
    the supported Codex, Claude, Cursor, Antigravity, Gemini, Grok, Pi, OpenCode,
    and VS Code stores.
  • Preserve native message/parent coordinates where available and retain source
    ordinals only when the scan direction proves them stable.
  • Add frontend-neutral ConversationUnit grouping with explicit native_tree,
    source_order, or unordered fidelity. It represents only the records
    observed by the caller; it does not claim source completeness, select a
    revision, or invent timestamp chronology.
  • Bound source-order linearization to one physical (store, adapter, path)
    domain, so file-local ordinals from separate transcripts cannot fabricate a
    shared timeline.
  • Replace raw ad hoc engine dedupe with seven hash-free, coordinate-aware tuple
    forms. Namespaced coordinates may compare across physical views; unnamespaced
    coordinates remain scoped to their source path and adapter.
  • Reuse prepared identity bundles at CLI/MCP/TUI boundaries, keep identity work
    off the Textual message pump, and bound/generation-gate the HUD caches.
  • Keep agref1: and agcur1: as opaque version 1 tokens. Positioned results
    bind the physical fingerprint to their occurrence coordinate; a position-blind
    ref retains bounded first-match fallback only while every other normalized
    fingerprint input still matches. Refs and cursors are snapshot-relative
    physical handles, while canonical IDs remain comparison handles.
  • Validate opaque refs and cursors as canonical base64url-encoded JSON, rejecting
    alternate spellings and JSON booleans at integer fields before dispatch.

ADR 0015 records the canonical envelopes, field policy, nullability, dedupe
forms, topology limits, privacy model, physical-handle lifetime, and downstream
contract for bookmarks, exports, and similarity.

TUI controls

  • Replace Textual's global command palette with one shared pi-style slash-command
    menu across HUD and grep-log.
  • Keep command input and focus visible through zoom, add clean /screenshot SVG
    capture, and make help, filtered-row styling, and quit confirmation consistent.

Issue refinement

The issue proposed several useful directions but left important policy forks
open. This PR resolves them conservatively:

  • Ship full purpose-specific content, occurrence, and thread IDs rather than a
    single short universal ID with auto-widening.
  • Exclude mutable provenance, model, timestamp, and path from content identity.
  • Do not derive flat-store thread IDs from paths or seed content.
  • Do not publish a Merkle root or imply that an observed thread is complete.
  • Keep engine equality and physical ref resolution as separate policies.
  • Keep the scan candidate path free of cryptographic hashing.

This gives #79 durable record/thread targets with an explicit content fallback,
requires #81 to own null-thread partitioning and complete-source/revision policy,
and lets #82 use canonical IDs for result/cache equality while using agref1:
to resolve a physical seed.

Compatibility and performance

  • Existing result fields remain compatible; the four identity fields are
    additive and required-nullable at typed output boundaries. agcur1: bytes and
    positionless agref1: bytes remain unchanged. Newly emitted positioned refs
    gain occurrence-aware fingerprints. Physical refs and cursors are
    snapshot-relative and may need to be reacquired after store or normalization
    changes.
  • Two exact-SHA, five-command benchmark matrices ran 25 samples per row in both
    commit orders. No user-visible command crossed the 5% median / 10% p95
    investigation threshold in both orders.
  • A 1 MiB identity takes about 0.65 ms on the measured machine. Prepared CLI and
    MCP consumers hash zero prompt bytes, and 500 threadless 1 MiB records are
    rejected before hashing.
  • The trusted source-order fast path improves its isolated 600,000-position
    benchmark by about 15% in both orders. End-to-end parsing remained order-noisy,
    so no broader speedup is claimed.
  • The follow-up Cursor positioned-iterator sample was effectively tied with the
    former iterator, at about 9.5 microseconds per loop.

Verification

The required repository gate passed on the final commit:

$ rm -rf docs/_build; uv run ruff check . --fix --show-fixes; uv run ruff format .; uv run ty check; uv run py.test --reruns 0 -vvv; just build-docs;

Results: 2,720 passed, 3 skipped, and 3 expected failures; documentation built
successfully.

The original three-way implementation bakeoff compared position-aware v1
fingerprints, canonical record IDs embedded in refs, and explicit v2 refs.
Adversarial Python, MCP/DX, and terminal/TUI judging selected the position-aware
v1 design 2-1: it resolves repeated turns exactly without longer handles or
coupling physical locators to canonical identity, while retaining a bounded
position-blind fallback.

A follow-up implementation bakeoff compared a localized physical-source repair,
a typed source-coordinate domain, and late identity/dedupe disambiguation.
Adversarial Python, typing, performance, MCP/DX, TUI, concision, and slop reviews
selected the localized repair: it fixes Cursor's within-line coordinates and
bounds conversation ordering for every adapter without expanding public models
or changing canonical IDs.

A live MCP smoke test resolved all seven positioned results from a bounded
ten-result page to their exact record IDs and rejected a malformed ref without
emitting store content.

Independent adversarial reviews also reproduced the source-domain, session,
ordering, threadless-hash, CLI/MCP parity, TUI worker/cache/race, ref-byte,
privacy, import-boundary, and documentation invariants. A watchdog-enabled live
TUI run searched 3,256 real-store matches, resized between 77 and 40 columns,
switched records, filtered to zero and back, and exited cleanly; full identity
handles stayed on one row at both widths.

Closes #80

@tony tony force-pushed the issue-80-deterministic-identities branch from 6e41258 to ce9b300 Compare July 12, 2026 11:06
@tony tony force-pushed the issue-80-deterministic-identities branch from ce9b300 to adcf5e0 Compare July 12, 2026 12:30
@tony tony force-pushed the issue-80-deterministic-identities branch from adcf5e0 to cb6377d Compare July 12, 2026 20:14
@tony tony force-pushed the issue-80-deterministic-identities branch from 4bb78dd to 88fa234 Compare July 12, 2026 23:16
@tony tony force-pushed the issue-80-deterministic-identities branch from 114a66f to 7b27c8f Compare July 12, 2026 23:41
@tony tony force-pushed the issue-80-deterministic-identities branch from f5d4530 to 06dfe7d Compare July 13, 2026 01:41
@tony tony force-pushed the issue-80-deterministic-identities branch from b4724de to 619a9af Compare July 13, 2026 23:05
@tony tony force-pushed the issue-80-deterministic-identities branch from 619a9af to eef6ae2 Compare July 13, 2026 23:51
@tony tony force-pushed the issue-80-deterministic-identities branch from eef6ae2 to c667bd6 Compare July 14, 2026 00:09
tony added 28 commits July 14, 2026 18:36
why: Unnamespaced thread coordinates are source-local, so treating
them as logical IDs collapsed records from separate stores and paths.

what:
- Add a fixed source-thread dedupe form for unnamespaced anchors.
- Keep logical and fallback forms exclusive to namespaced anchors.
- Compare cheap inventory metadata before exact record text.
- Pin driver, bounded-scan, ordering, and ADR behavior with tests.
why: Source-order adapters pass enumerate() ordinals through the
coordinate validator. Normalizing two absent native coordinates per
record adds measurable work to large histories.

what:
- Return the same source-order position before native normalization
  when both native coordinates are absent.
- Cover the fast path while keeping validation tests for native and
  malformed coordinates.
why: Exhaustive adapter and narrow-terminal permutations exposed
explicit source anchors that remained null and HUD identity rows that
wrapped or cropped after resize.

what:
- Preserve defensible thread and occurrence coordinates across default
  and inspectable adapters while keeping ambiguous fallbacks null.
- Keep full HUD handles on one row with responsive labels and a
  resize-only header refresh.
- Add regressions for adapter topology, narrow rendering, and detail
  find row accounting.
why: Distinct stored occurrences could share a position-blind physical
fingerprint, so inspect_result stopped at the first match and returned
the wrong canonical occurrence.

what:
- Fold validated native or source-order coordinates into current search
  ref fingerprints without changing token shape or length.
- Match historical position-blind fingerprints as a bounded fallback.
- Document compatibility and cover repeated, threadless, and malformed
  coordinate cases.
why: Permissive base64 decoding discarded non-alphabet characters, and
Python's integer hierarchy let JSON booleans pass as token versions,
offsets, and limits.

what:
- Decode refs and cursors with CPython's validating base64url path.
- Reject booleans at every JSON integer boundary.
- Cover malformed refs and both cursor kinds while preserving valid
  version 1 bytes.
why: Cursor transcripts can store identical turns at different raw
positions. Adapter-level content deduplication erased that occurrence
distinction before deterministic record IDs were assigned.

what:
- Retain every readable Cursor transcript occurrence.
- Let raw source ordinals distinguish otherwise equal stored turns.
why: Latest catalog semantics make encrypted loose Antigravity protobuf
stores discoverable only as catalog entries. They cannot supply readable
records or deterministic thread anchors.

what:
- Limit Antigravity UUID anchors to readable CLI stores.
- Test valid conversation identity and catalog-only rejection.
- Keep duplicate-view coverage on the three readable CLI views.
why: The identity changelog contract checks its reader-facing promise
without normalizing Markdown line breaks. The refreshed release prose split
that promise across lines.

what:
- Keep the repeated-content phrase intact in the release entry.
why: Release rollover moved issue 80 from the a34 placeholder into the a35
placeholder. Pinning a changelog test to one prerelease number makes every
release commit fail an otherwise current feature branch.

what:
- Locate the active Yet to be released block by its versioned heading.
- Check the lead sentence against the captured current version.
why: Textual’s built-in Ctrl-P palette competes with the agentgrep
shell and can replace the active screen and focus state.

what:
- Disable Textual’s command palette and clear its provider set.
- Prove Ctrl-P is inert in stacked and split HUD layouts.
- Remove stale command-palette wording from TUI docs and comments.
why: Removing the command palette left key help and theme switching
without a common, layout-safe entry point. A shared slash dispatcher
keeps recognized commands out of search while preserving literal
slash-text compatibility.

what:
- Add common keys and argument-aware theme commands with boolean
  handlers.
- Share exact dispatch and extension commands across HUD and grep-log.
- Cover command metadata, menu routing, validation, and mounted
  behavior.
why: Textual screen maximization hides the command shell and makes pane
zoom hard to reverse. Layout-owned zoom keeps commands available while
preserving responsive geometry.

what:
- Add layout-aware maximize and minimize slash commands.
- Zoom HUD columns and grep-log content with recoverable classes.
- Cover selection, focus, geometry, refusal, and recovery behavior.
why: A screenshot command needs to exclude transient input and
completion chrome while stale callbacks must not capture replacement or
detached layouts.

what:
- Add a no-argument screenshot command using Textual delivery after
  refresh.
- Guard deferred delivery by attachment and active-screen identity.
- Cover clean capture, preserved state, teardown, and pump
  classification.
why: Filtered-out rows kept old-theme cache entries and resurfaced with
mixed colors after a filter widened. Palette-aware keys preserve correct
rendering without clearing an unbounded cache on the message pump.

what:
- Key rendered rows by theme and record identity.
- Keep visible repaint work on the existing bounded chunk path.
- Cover hide, theme-switch, and filter-widen behavior.
why: GrepLog could execute slash commands only when fully typed, but
it offered no menu, leaving the shared surface undiscoverable after a
layout switch.

what:
- Move slash-menu state and selection into the shared layout shell.
- Mount and drive the same completion dropdown in GrepLog.
- Cover narrow prefix selection and command parity across layouts.
- Refresh the HUD menu expectation for the complete command set.
why: The character cap prevents oversized records from freezing Textual,
but users need to know that the view is bounded and how to retrieve the
unchanged full record.

what:
- State the line and character limits in the detail renderer contract.
- Document CLI JSON and MCP inspection routes to the full body.
why: The palette replacement is discoverable in-app, but the TUI guide
still omitted its shared commands, layout-specific zoom targets, and
screenshot delivery behavior.

what:
- Document command-menu discovery in HUD and GrepLog.
- List shared commands and their argument forms.
- Explain key-panel toggling and clean screenshot delivery.
why: Each `/keys` call created a uniquely identified Toast, so repeated
calls stacked panels with no keyboard command to close them.

what:
- Toggle Textual's singleton key-help panel from the shared command.
- Keep HUD and GrepLog input, results, focus, and zoom state intact.
- Cover open, close, notification absence, and continued typing.
why: Directional navigation ignored logical zoom, so focus could enter a
hidden sibling pane and make later keystrokes appear unresponsive.

what:
- Switch the logical zoom before focusing a sibling-pane widget.
- Preserve the visible search shell and existing focus graph.
- Cover wide and stacked navigation in both zoom directions.
why: A q binding bypassed the staged Ctrl-C warning and exited directly
from results. Grep-log had the same one-key exit path.

what:
- Share keyed quit confirmation across both TUI layouts.
- Cancel pending confirmation on focus, mode, and workflow changes.
- Cover results, detail, grep-log, input, and cancellation paths.
why: Draining scans cannot use source-local dedupe counts to stop early.
Identity-aware keys were therefore computed in the scanner and again in
the global collector.

what:
- Allocate local dedupe state only for bounded scans with a limit.
- Keep key projection at the global frontier for draining scans.
why: Adapter helpers return fresh mutable candidates. Copying each one
solely to attach source identity wastes allocations on every parsed row.

what:
- Attach namespaces and positions directly to fresh candidates.
- Verify normalization receives the original enriched candidate.
why: Optimized JSONL scans discarded physical ordinals, so the same
native-free occurrence could lose its record ID and repeated equal turns
could collapse during deduplication.

what:
- Carry byte-derived source coordinates through forward, filtered, and
  reverse JSONL readers.
- Apply stable positions across all optimized adapters and document gapped
  source ordinals.
- Cover scan parity, multi-turn lines, exact boundary spans, and bounded
  execution.
why: Cursor JSONL events can contain multiple message candidates.
Reusing the raw line index for each candidate collapsed repeated
occurrences into the same record ID.

what:
- Read Cursor transcripts with the positioned JSONL iterator.
- Combine each byte offset with its within-line candidate index.
- Cover stable repeated IDs, dedupe, and single-pass parsing.
why: JSONL ordinals are file-local. Comparing them across paths
fabricated chronology for records that shared only a thread and
adapter.

what:
- Include the physical path in the private ordering domain.
- Withhold linearization across separate source files.
- Document that physical topology stays outside canonical IDs.
why: The position-blind fallback only works while every other
normalized fingerprint field still matches. A blanket persisted-ref
promise overstated this prerelease contract.

what:
- Describe refs and cursors as snapshot-relative physical handles.
- Keep the narrow position-blind first-match fallback.
- Direct callers to canonical IDs and fresh refs after changes.
why: The branch already carries a material pi-like TUI control pass.
The release entry described only deterministic identity.

what:
- Name both release outcomes in the lead.
- Add a TUI command, focus, screenshot, and quit deliverable.
- Keep the two-deliverable release contract under test.
why: Trunk now reserves version-led changelog summaries for release cuts. Feature branches should contribute only scoped deliverable sections.

what:
- Remove the unreleased a36 summary while retaining both deliverables.
- Move the identity ADR link into its deliverable and guard the rule.
@tony tony force-pushed the issue-80-deterministic-identities branch from 5ffd718 to 4e49bbd Compare July 14, 2026 23:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deterministic IDs for conversations / prompts

1 participant