tests: pin the pre-recall prompt-cache freeze - #268
Conversation
AgentEngine._get_or_create_client freezes a session's first successful
pre-recall in session metadata (engine.py:1149) so every later rebuild
renders a byte-identical system prompt and hits the prompt cache. Nothing
tested it, in either direction: `git grep -c recalled_memories origin/main
-- tests/` and the same for meta_updates both return 0 hits in any test
file, and tests/test_prompts.py never passes recalled_memories at all.
Deleting engine.py:1149 leaves the whole suite green, and nothing else
re-derives the value, so the regression would be silent - every rebuild
re-recalls live, a session's priors drift, and the cache hit the freeze
exists to buy is lost with no test failing.
The gap survived because the payoff is a cross-rebuild property. Dropping
the write leaves an observably identical first turn; the loss shows only on
a later rebuild of the same session, and no existing test spanned two
rebuilds.
Test-only and insert-only. TestPreRecallFreeze goes in the file that
already drives _get_or_create_client end to end against the real db fixture
with a local stub backend, so no production line changes and no new seam is
needed - the stub's create_client keeps the SessionSpec it is already
handed. Phase A asserts the metadata now holds the priors AND that the
first rendered system prompt carries them (prompts.py:166-168, previously
unpinned in tests/ and in test_prompts.py). Phase B evicts the client via
sessions.remove_client, rearms recall with different priors, calls again,
and asserts a second create_client really happened before checking that the
second prompt carries the ORIGINAL priors and none of the rearmed ones. The
precondition is not optional: a live client returns at engine.py:1077
before session_meta is parsed, so without the eviction the second call
never re-enters the freeze block and both assertions would be satisfied by
the first call's side effects.
Coverage-only, so discrimination is the deliverable. Eight mutants, all
killed, each arm asserting its mutation applied and restoring the tree
afterwards, with the unmutated control green at both ends of the matrix:
deleting engine.py:1149 (phase A metadata); frozen_recall = None so every
rebuild re-recalls (phase B, second prompt carried the rearmed priors);
freezing the wrong value, = [] (phase A); the render call dropping the
priors (phase A prompt only); the frozen-read branch reading then
discarding (phase B prompt only); reversing the frozen list and doubling
it, each of which changes the prompt bytes while leaving membership,
exclusion, await_count and the store all satisfied (phase B byte-identity
only); and removing the eviction from the test itself, which fails loudly
at the precondition (assert 1 == 2) rather than passing silently.
Weakening phase A to the key-presence-plus-await_count shape makes the
wrong-value mutant survive phase A - the second-prompt assertion is what
kills it - so the store check, the first-prompt check, the second-prompt
check and the byte-identity check each uniquely kill a different mutant
and none is decoration.
"Reused verbatim" is byte-identity, and membership is strictly weaker:
prompts.py:167 renders "\n".join(f"- {m}" for m in recalled_memories), so
order and multiplicity are prompt bytes, and byte-identity is exactly the
property the prompt cache needs. A reordered or duplicated frozen list
therefore satisfies every membership, exclusion, await_count and store
assertion while destroying the payoff; the reversal and duplication
mutants both survive without the byte-identity check and both die on it,
at that assertion specifically. It is ADDITIVE rather than a replacement:
under the mutant where the render call drops the priors entirely, both
prompts are equally wrong, so comparing them to each other passes and the
first-prompt membership check is that mutant's only observer. Substituting
would have traded one kill for another.
Target file 20 -> 21 passed. Full suite compared by failure NAME, not
count: 7 failed / 2933 passed before, 7 failed / 2934 passed after, the two
sorted name sets byte-identical, all 7 pre-existing and none in a file this
touches. The new test is 20/20 stable and clean in one process with
test_memorize_background, test_session_context_tool and test_prompts.
ruff reports one F401 at tests/test_engine_backend_selection.py:196; it is
pre-existing (the identical error is reported on the untouched origin/main
blob at the same line) and deliberately left alone as an unrelated concern.
Internal second-model review (2 rounds, 4 findings agreed and fixed, 0 disagreed)Before opening this PR I ran it through an independent review pass: a cold read of the Round 1 - one major, co-found by both reviewersAssertions were weaker than the stated contract. The class docstring promises the priors Confirmed mechanically rather than argued: rendering The remedy was prototyped before being prescribed, and the measurement changed it. The Soundness was pre-verified so the fix could not be flaky: the only non-deterministic input to Round 2 - artifact accuracyTwo figures in this PR description were stale. Adding the two new mutants made "six Noted, not blocking
Recorded elsewhere, deliberately not raised hereWhile reading the eviction path I noticed The |
Pre-PR validation gate (a-i)
Also measured: |
Symptom
AgentEngine._get_or_create_clientfreezes a session's first successful pre-recall(
engine.py:1149) so every later rebuild renders a byte-identical system prompt and hitsthe prompt cache. Nothing tested it, in either direction:
Delete
engine.py:1149and the whole suite stays green. Nothing else re-derives the value,so the regression is silent: every rebuild re-recalls live, a session's priors drift, and
the cache hit the freeze exists to buy is gone with no test failing.
Root cause of the gap
The freeze's payoff is a cross-rebuild property. Dropping the write leaves an observably
identical first turn; the loss only shows on a later rebuild of the same session. No
existing test spanned two rebuilds, so the line entered the tree untested.
Fix
Test-only, insert-only: one class in the file that already drives
_get_or_create_clientend to end against the real
dbfixture with a local stub backend. No production linechanges and no new seam - the stub's
create_clientjust keeps theSessionSpecit isalready handed.
P1: metadata holdsrecalled_memories == P1,and the rendered system prompt carries each prior of
P1(prompts.py:166-168), which wasunpinned in
tests/and intest_prompts.py.sessions.remove_client), rearm recall toP2, callagain: assert the rebuild really happened (a live client returns at
engine.py:1077before metadata is parsed, so without this precondition the arm is vacuous), the second
prompt carries
P1and noP2, andrecall.await_count == 1.Tests
Coverage-only, so discrimination is the deliverable: eight mutants, each killed, each arm
asserting the mutation applied and restoring the tree afterwards, with the unmutated control
green at both ends. The metadata check, the first-prompt check and the second-prompt check
each uniquely kill a different mutant, so no assertion is decoration. Byte-identity is
the only check that sees order and multiplicity: reversing or doubling the frozen list
passes every other assertion, and both mutants survive without it.
Mutation matrix
engine.py:1149frozen_recall = None- rebuild re-recallsP2)= [])recalled_memories=None)1 == 2), not a silent passTarget file 20 -> 21 passed. Full suite compared by failure name: 7 failed / 2933 passed
-> 7 failed / 2934 passed, the two sorted name sets byte-identical (all 7 pre-existing, none
in a file this touches). New test 20/20 stable.
No related open issue found.