fix(observer): advance coverage watermark when observer records nothing#24
Open
joakimp wants to merge 1 commit into
Open
fix(observer): advance coverage watermark when observer records nothing#24joakimp wants to merge 1 commit into
joakimp wants to merge 1 commit into
Conversation
The observer fires whenever rawTokensSinceObservationCoverage >=
observeAfterTokens. On a non-empty result it appends an
om.observations.recorded marker, which advances the watermark. But when the
observer returns no observations, runObserverStage only notified and returned
"continue" without writing any coverage marker.
Because om.observations.recorded is the only thing that advanced observation
coverage (isValidCoverageEntry requires a non-empty observations array), a
single "nothing worth recording" verdict on a >= observeAfterTokens chunk left
the watermark stuck. The same span (growing by the new tail each turn) stayed
over the threshold, so the observer re-ran on every agent_start/turn_end,
re-emitting "observer returned no observations" and burning an observer model
call each time, until some later turn happened to yield an observation.
Fix: introduce a coverage-only marker om.observations.none { coversUpToId }.
On an empty observer result we append it; observation-coverage scans
(rawTokensSinceObservationCoverage and the observer chunk-start index) now
consider both om.observations.recorded and om.observations.none via
OBSERVATION_COVERAGE_TYPES. The marker carries no observations, so projection,
fold, reflector and dropper ignore it; reflection and drop watermarks are
unchanged (they still track their own marker types only). Raw entries remain
available to the reflector and until compaction, so no memory is lost.
Also downgrades the per-empty notification from "warning"
("observer returned no observations") to an "info"
("observer found nothing new to record"), since an empty verdict is now a
normal, one-shot event rather than a recurring error condition.
Tests: update the no-output consolidation test to assert the new coverage-only
marker contract; add progress-helper tests that a none marker advances
observation coverage without a recorded marker and does not touch
reflection/drop coverage. Full suite: 181 passing.
Owner
|
This is not a good idea, usually if the observer fails to extract any observations is because of technical errors, advancing the watermark will create a gap in the observed session. in the rare scenario the observer genuinely didn't have anything to observe, the next observer will have more meat/context to generate rich, useful observations. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #23
Summary
When the observer returns no observations for a chunk, the observation
coverage watermark is never advanced, so the observer re-runs on every
subsequent turn over the same (growing) span — spamming
Observational memory: observer returned no observationsand burning oneobserver model call per turn — until some later turn happens to produce an
observation.
This PR adds a coverage-only ledger marker (
om.observations.none) that theobserver writes on an empty result, so the watermark advances and the span is
not reprocessed. No observations are fabricated; no memory is lost.
Problem (observed in the wild)
With
observational-memory.debugLog: true, a single session showed the observerfiring 9 times: 5 of them empty, back-to-back. The empty runs kept the chunk
start pinned while only the end grew, so token count climbed past the trigger
every turn:
Every
agent_start/turn_endduring that window re-ran the observer on the sameleading span and re-emitted the warning. It only recovered once enough new tail
accumulated that the observer incidentally found something to record.
Root cause
runObserverStage(insrc/hooks/consolidation-trigger.ts) gates onrawTokensSinceObservationCoverage(entries) >= observeAfterTokens. On anon-empty result it appends an
om.observations.recordedmarker, which advancesthe watermark. The empty branch did not:
And
om.observations.recordedis the only entry type that advancesobservation coverage —
isValidCoverageEntry(insession-ledger/progress.ts)requires a non-empty
observationsarray. So the empty verdict left thewatermark stuck and the chunk perpetually over-threshold.
Fix
OM_OBSERVATIONS_NONE = "om.observations.none"withdata
{ coversUpToId }(session-ledger/types.ts): builderbuildObservationsNoneData, guardisObservationsNoneEntry, added toV3MemoryCustomType.OBSERVATION_COVERAGE_TYPES = [OM_OBSERVATIONS_RECORDED, OM_OBSERVATIONS_NONE].latestCoverageIndex/latestCoverageMarkerIdaccept one type or an array;rawTokensSinceObservationCoverageand the observer chunk-start index use thearray.
om.observations.nonemarker for thechunk it just processed, then returns.
warning(
observer returned no observations) toinfo(
observer found nothing new to record), since with the watermark advancingthis is a normal one-shot event, not a recurring error.
The marker carries no observations, so
projection,fold, the reflector andthe dropper ignore it (they key off the typed
isObservations*Entryguards).Reflection and drop watermarks are intentionally unchanged — they still
track only their own marker types. Raw entries remain available to the reflector
and until compaction, so nothing is lost; the observer simply respects its own
"nothing to record here" verdict instead of repeating it every turn.
Alternatives considered
om.observations.recordedto allow an emptyobservationsarray.Rejected: it would make the "valid coverage" and "has observations" predicates
disagree for the same entry type, and an
observations.recordedentry thatrecords nothing is misleading. A dedicated marker is clearer and keeps the
projection/fold guards untouched.
observeAfterTokens. Only reduces frequency; the re-fire loopstill occurs on any genuinely low-salience span.
Backward compatibility
Additive. Old ledgers without
om.observations.nonemarkers behave exactly asbefore. Unknown custom types are already ignored by
fold/projection/view(no exhaustive
customTypeswitch), so a downgrade after this lands degradesgracefully (an
om.observations.noneentry is simply skipped, reverting to theprior — buggy — behavior for that one span).
Testing
npm run typecheck— clean.npm test— 181 passing (179 prior + 2 new).tests/consolidation-trigger.test.ts: the no-output case now asserts asingle
om.observations.none { coversUpToId }marker is appended (and noom.observations.recorded), reflector/dropper are not called, and thenotification is
info.tests/session-ledger-progress.test.ts: anonemarker advancesobservation coverage even with no recorded marker present, new source after it
is still uncovered, and it does not advance reflection/drop coverage.
observationsNoneEntryfixture +V3_OBSERVATIONS_NONEconstant.