Skip to content

tests: pin the pre-recall prompt-cache freeze - #268

Merged
pufit merged 1 commit into
ClickHouse:mainfrom
oranjeai:oranjeai/pin-prerecall-freeze
Aug 4, 2026
Merged

tests: pin the pre-recall prompt-cache freeze#268
pufit merged 1 commit into
ClickHouse:mainfrom
oranjeai:oranjeai/pin-prerecall-freeze

Conversation

@oranjeai

@oranjeai oranjeai commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Symptom

AgentEngine._get_or_create_client freezes a session's first successful pre-recall
(engine.py:1149) so every later rebuild renders a byte-identical system prompt and hits
the prompt cache. Nothing tested it, in either direction:

git grep -c 'recalled_memories' origin/main -- tests/   # 0 hits, any test file
git grep -c 'meta_updates'      origin/main -- tests/   # 0
git show origin/main:tests/test_prompts.py | grep -c recalled_memories   # 0

Delete engine.py:1149 and the whole suite stays green. Nothing else re-derives the value,
so the regression is silent: every rebuild re-recalls live, a session's priors drift, and
the cache hit the freeze exists to buy is gone with no test failing.

Root cause of the gap

The freeze's payoff is a cross-rebuild property. Dropping the write leaves an observably
identical first turn; the loss only shows on a later rebuild of the same session. No
existing test spanned two rebuilds, so the line entered the tree untested.

Fix

Test-only, insert-only: one class in the file that already drives _get_or_create_client
end to end against the real db fixture with a local stub backend. No production line
changes and no new seam - the stub's create_client just keeps the SessionSpec it is
already handed.

  • Phase A - first call, recall returns P1: metadata holds recalled_memories == P1,
    and the rendered system prompt carries each prior of P1 (prompts.py:166-168), which was
    unpinned in tests/ and in test_prompts.py.
  • Phase B - evict the client (sessions.remove_client), rearm recall to P2, call
    again: assert the rebuild really happened (a live client returns at engine.py:1077
    before metadata is parsed, so without this precondition the arm is vacuous), the second
    prompt carries P1 and no P2, and recall.await_count == 1.

Tests

Coverage-only, so discrimination is the deliverable: eight mutants, each killed, each arm
asserting the mutation applied and restoring the tree afterwards, with the unmutated control
green at both ends. The metadata check, the first-prompt check and the second-prompt check
each uniquely kill a different mutant, so no assertion is decoration. Byte-identity is
the only check that sees order and multiplicity: reversing or doubling the frozen list
passes every other assertion, and both mutants survive without it.

Mutation matrix
mutant result
delete engine.py:1149 FAIL (phase A metadata)
frozen_recall = None - rebuild re-recalls FAIL (phase B: prompt carried P2)
freeze the wrong value (= []) FAIL (phase A)
render call drops the priors (recalled_memories=None) FAIL (phase A prompt only)
frozen-read branch reads then discards FAIL (phase B prompt only)
eviction removed from the test FAIL loudly at the precondition (1 == 2), not a silent pass
frozen list reversed (order) FAIL (phase B byte-identity only; survives without it)
frozen list doubled (multiplicity) FAIL (phase B byte-identity only; survives without it)

Target file 20 -> 21 passed. Full suite compared by failure name: 7 failed / 2933 passed
-> 7 failed / 2934 passed, the two sorted name sets byte-identical (all 7 pre-existing, none
in a file this touches). New test 20/20 stable.

No related open issue found.

AgentEngine._get_or_create_client freezes a session's first successful
pre-recall in session metadata (engine.py:1149) so every later rebuild
renders a byte-identical system prompt and hits the prompt cache. Nothing
tested it, in either direction: `git grep -c recalled_memories origin/main
-- tests/` and the same for meta_updates both return 0 hits in any test
file, and tests/test_prompts.py never passes recalled_memories at all.
Deleting engine.py:1149 leaves the whole suite green, and nothing else
re-derives the value, so the regression would be silent - every rebuild
re-recalls live, a session's priors drift, and the cache hit the freeze
exists to buy is lost with no test failing.

The gap survived because the payoff is a cross-rebuild property. Dropping
the write leaves an observably identical first turn; the loss shows only on
a later rebuild of the same session, and no existing test spanned two
rebuilds.

Test-only and insert-only. TestPreRecallFreeze goes in the file that
already drives _get_or_create_client end to end against the real db fixture
with a local stub backend, so no production line changes and no new seam is
needed - the stub's create_client keeps the SessionSpec it is already
handed. Phase A asserts the metadata now holds the priors AND that the
first rendered system prompt carries them (prompts.py:166-168, previously
unpinned in tests/ and in test_prompts.py). Phase B evicts the client via
sessions.remove_client, rearms recall with different priors, calls again,
and asserts a second create_client really happened before checking that the
second prompt carries the ORIGINAL priors and none of the rearmed ones. The
precondition is not optional: a live client returns at engine.py:1077
before session_meta is parsed, so without the eviction the second call
never re-enters the freeze block and both assertions would be satisfied by
the first call's side effects.

Coverage-only, so discrimination is the deliverable. Eight mutants, all
killed, each arm asserting its mutation applied and restoring the tree
afterwards, with the unmutated control green at both ends of the matrix:
deleting engine.py:1149 (phase A metadata); frozen_recall = None so every
rebuild re-recalls (phase B, second prompt carried the rearmed priors);
freezing the wrong value, = [] (phase A); the render call dropping the
priors (phase A prompt only); the frozen-read branch reading then
discarding (phase B prompt only); reversing the frozen list and doubling
it, each of which changes the prompt bytes while leaving membership,
exclusion, await_count and the store all satisfied (phase B byte-identity
only); and removing the eviction from the test itself, which fails loudly
at the precondition (assert 1 == 2) rather than passing silently.
Weakening phase A to the key-presence-plus-await_count shape makes the
wrong-value mutant survive phase A - the second-prompt assertion is what
kills it - so the store check, the first-prompt check, the second-prompt
check and the byte-identity check each uniquely kill a different mutant
and none is decoration.

"Reused verbatim" is byte-identity, and membership is strictly weaker:
prompts.py:167 renders "\n".join(f"- {m}" for m in recalled_memories), so
order and multiplicity are prompt bytes, and byte-identity is exactly the
property the prompt cache needs. A reordered or duplicated frozen list
therefore satisfies every membership, exclusion, await_count and store
assertion while destroying the payoff; the reversal and duplication
mutants both survive without the byte-identity check and both die on it,
at that assertion specifically. It is ADDITIVE rather than a replacement:
under the mutant where the render call drops the priors entirely, both
prompts are equally wrong, so comparing them to each other passes and the
first-prompt membership check is that mutant's only observer. Substituting
would have traded one kill for another.

Target file 20 -> 21 passed. Full suite compared by failure NAME, not
count: 7 failed / 2933 passed before, 7 failed / 2934 passed after, the two
sorted name sets byte-identical, all 7 pre-existing and none in a file this
touches. The new test is 20/20 stable and clean in one process with
test_memorize_background, test_session_context_tool and test_prompts.

ruff reports one F401 at tests/test_engine_backend_selection.py:196; it is
pre-existing (the identical error is reported on the untouched origin/main
blob at the same line) and deliberately left alone as an unrelated concern.
@oranjeai

oranjeai commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author
Internal second-model review (2 rounds, 4 findings agreed and fixed, 0 disagreed)

Before opening this PR I ran it through an independent review pass: a cold read of the
resulting code by a reviewer that never wrote it, plus a separate second model reviewing the
diff in isolation. Two rounds. Every finding below was agreed and fixed; nothing was
overridden.

Round 1 - one major, co-found by both reviewers

Assertions were weaker than the stated contract. The class docstring promises the priors
are reused verbatim on every rebuild, and engine.py:1131-1136 says the point is
byte-identical system prompts. But phase B only asserted membership of the first list,
absence of the rearmed list, await_count == 1, and the stored value. Membership is strictly
weaker than byte-identity, and prompts.py:167 renders "\n".join(f"- {m}" ...), so order
and multiplicity are prompt bytes.

Confirmed mechanically rather than argued: rendering list(reversed(first)) and first + first each produce different prompt bytes and passed every assertion the test then had.
The one property the freeze exists to buy was the one property nothing observed.

The remedy was prototyped before being prescribed, and the measurement changed it. The
obvious form - replacing the membership loops with prompts[1] == prompts[0] - is
insufficient on its own: under a mutant where the render call drops the priors entirely,
both prompts are equally wrong, so that equality holds and passes. Substituting would have
traded away an existing kill. It had to be additive, which is what shipped.

Soundness was pre-verified so the fix could not be flaky: the only non-deterministic input to
build_system_prompt is the Session Context line, which is deliberately date-resolution only
(prompts.py:139-146); skills are absent on a bare engine; the tool registry is memoized.

Round 2 - artifact accuracy

Two figures in this PR description were stale. Adding the two new mutants made "six
mutants" and the six-row matrix table wrong; the true count is eight. Both understated
coverage, so neither was a merge hazard - but the commit message already said eight, so the
PR would have contradicted itself across surfaces. The count was checked three independent
ways and required to agree before the description was corrected and re-reviewed from scratch.

Noted, not blocking

  • The new comment cites prompts.py:167 by line number. Correct today, but it is the only
    bare line-number citation in the repo (0 on main), and line numbers rot. The comment
    states the mechanism, so a stale number costs nothing semantically.
  • A first recall that legitimately returns nothing freezes [] and later renders no
    section. Coherent pre-existing behaviour, outside a test-only change's scope.

Recorded elsewhere, deliberately not raised here

While reading the eviction path I noticed SessionManager.remove_client (sessions.py:347)
pops _client_locks[session_id] while _get_or_create_client is holding that same lock
(engine.py:1027), so a concurrent waiter re-enters get_lock and receives a fresh
asyncio.Lock - exclusion is not actually held across the recreate. Pre-existing, unrelated
to a test-only change, and not something to fix in this PR. Flagging it for whoever next
touches that path.

The F401 on types.SimpleNamespace at :196 is pre-existing: the identical error is
present on the untouched main blob at the same line.

@oranjeai

oranjeai commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author
Pre-PR validation gate (a-i)
# Question Answer
a Deterministic repro? Yes, 1/1. This is a coverage gap, so the reproduced "failure" is the absence of a failing test: delete nerve/agent/engine.py:1149, then pytest tests/test_engine_backend_selection.py::TestPreRecallFreeze FAILS with this change present and the whole suite PASSES without it. Not probabilistic.
b Root cause explained? Yes. The freeze's payoff is a cross-rebuild property - dropping the write leaves an observably identical first turn, and the loss shows only on a later rebuild of the same session. No test spanned two rebuilds, so the line entered the tree untested.
c Fix matches root cause? Yes. The gap is "no test spans two rebuilds"; the change is a two-phase test that spans two rebuilds and observes the store, the render, and the byte-identity of the rendered prompt across rebuilds. Nothing widened, tagged or guarded.
d Test intent preserved / new tests added? Yes. Insert-only (git diff --numstat origin/main...HEAD = 98 0): no existing test modified, weakened, reordered or removed. The new test IS the added coverage.
e Demonstrated in both directions? Yes. Eight mutants, all killed, control green at both ends of the matrix. Two of them (reversing the frozen list, doubling it) exist to pin the contract's own word: prompts.py:167 renders "\n".join(f"- {m}" ...), so order and multiplicity are prompt bytes, and each of those mutants satisfies every membership, exclusion, await_count and store assertion while changing the prompt bytes. Both were measured in BOTH directions - they SURVIVE without the byte-identity assertion and are KILLED at that assertion with it. Additionally, weakening phase A to a key-presence-plus-await_count design makes the wrong-value mutant survivable through phase A - the second-prompt assertion is what kills it, so the stronger assertions are load-bearing rather than cosmetic.
f General across code paths? Every carrier of the invariant is pinned or dispositioned: the write (phase A), the frozen-read branch (phase B, both content and byte-identity), the render call (phase A prompt), the persist (transitively, via the real store). The failure path (recall raises, no write) is deliberately left to the in-flight PR that adds its own arm for it - recorded, not skipped. The three sibling callers delegate into this method and are not carriers.
g Generalizes across inputs? Two distinct multi-element prior lists are used, disjoint by construction, so "reused verbatim" cannot pass by coincidence; the empty-list case is itself a killed mutant, and the reordered and duplicated cases are two more.
h Backward compatible? Vacuously - no production line changes, so no setting default, no SettingsChangesHistory, no serialization format, no migration.
i Invariants and contracts preserved? No production contract is touched. The test mutates no module global, uses the per-test db fixture, and its stub implements exactly the backend surface this method calls. Verified free of cross-suite contamination by running it in one process with test_memorize_background, test_session_context_tool and test_prompts (47 passed).

Also measured: ruff check on the changed file reports one F401 at line 196, which is
pre-existing - the identical error is reported on the untouched origin/main blob at the
same line. Left alone as an unrelated concern. (This repo's CI runs pytest tests/ -v plus a
Vite build; there is no lint job.)

@pufit
pufit merged commit a62cebe into ClickHouse:main Aug 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants