Skip to content

fix(observer): advance coverage watermark when observer records nothing#24

Open
joakimp wants to merge 1 commit into
elpapi42:masterfrom
joakimp:fix/observer-empty-coverage-watermark
Open

fix(observer): advance coverage watermark when observer records nothing#24
joakimp wants to merge 1 commit into
elpapi42:masterfrom
joakimp:fix/observer-empty-coverage-watermark

Conversation

@joakimp

@joakimp joakimp commented Jun 20, 2026

Copy link
Copy Markdown

Fixes #23

Summary

When the observer returns no observations for a chunk, the observation
coverage watermark is never advanced, so the observer re-runs on every
subsequent turn over the same (growing) span — spamming
Observational memory: observer returned no observations and burning one
observer model call per turn — until some later turn happens to produce an
observation.

This PR adds a coverage-only ledger marker (om.observations.none) that the
observer writes on an empty result, so the watermark advances and the span is
not reprocessed. No observations are fabricated; no memory is lost.

Problem (observed in the wild)

With observational-memory.debugLog: true, a single session showed the observer
firing 9 times: 5 of them empty, back-to-back. The empty runs kept the chunk
start pinned while only the end grew, so token count climbed past the trigger
every turn:

21:00:52 START first=1ffebad0 last=bd84d836 n=34 tok=10286  -> EMPTY (warning)
21:01:21 START first=1ffebad0 last=c87a1d5b n=36 tok=10995  -> EMPTY (warning)
21:01:40 START first=1ffebad0 last=f284f95c n=38 tok=11894  -> EMPTY (warning)
21:01:56 START first=1ffebad0 last=a606b101 n=40 tok=12423  -> EMPTY (warning)
21:02:19 START first=1ffebad0 last=dab0235d n=42 tok=12932  -> EMPTY (warning)
21:02:30 START first=1ffebad0 last=74daaf4f n=43 tok=13306  -> records 8  (watermark finally advances)

Every agent_start/turn_end during that window re-ran the observer on the same
leading span and re-emitted the warning. It only recovered once enough new tail
accumulated that the observer incidentally found something to record.

Root cause

runObserverStage (in src/hooks/consolidation-trigger.ts) gates on
rawTokensSinceObservationCoverage(entries) >= observeAfterTokens. On a
non-empty result it appends an om.observations.recorded marker, which advances
the watermark. The empty branch did not:

if (!observations || observations.length === 0) {
    debugLog("observer.empty", { coversUpToId });
    if (ctx.hasUI) ctx.ui?.notify("Observational memory: observer returned no observations", "warning");
    return "continue";          // <-- no coverage marker written
}

And om.observations.recorded is the only entry type that advances
observation coverage — isValidCoverageEntry (in session-ledger/progress.ts)
requires a non-empty observations array. So the empty verdict left the
watermark stuck and the chunk perpetually over-threshold.

Fix

  • New coverage-only marker OM_OBSERVATIONS_NONE = "om.observations.none" with
    data { coversUpToId } (session-ledger/types.ts): builder
    buildObservationsNoneData, guard isObservationsNoneEntry, added to
    V3MemoryCustomType.
  • Observation-coverage scans now consider both marker types via
    OBSERVATION_COVERAGE_TYPES = [OM_OBSERVATIONS_RECORDED, OM_OBSERVATIONS_NONE].
    latestCoverageIndex / latestCoverageMarkerId accept one type or an array;
    rawTokensSinceObservationCoverage and the observer chunk-start index use the
    array.
  • The empty observer branch appends an om.observations.none marker for the
    chunk it just processed, then returns.
  • The per-empty notification is downgraded from warning
    (observer returned no observations) to info
    (observer found nothing new to record), since with the watermark advancing
    this is a normal one-shot event, not a recurring error.

The marker carries no observations, so projection, fold, the reflector and
the dropper ignore it (they key off the typed isObservations*Entry guards).
Reflection and drop watermarks are intentionally unchanged — they still
track only their own marker types. Raw entries remain available to the reflector
and until compaction, so nothing is lost; the observer simply respects its own
"nothing to record here" verdict instead of repeating it every turn.

Alternatives considered

  • Relax om.observations.recorded to allow an empty observations array.
    Rejected: it would make the "valid coverage" and "has observations" predicates
    disagree for the same entry type, and an observations.recorded entry that
    records nothing is misleading. A dedicated marker is clearer and keeps the
    projection/fold guards untouched.
  • Just raise observeAfterTokens. Only reduces frequency; the re-fire loop
    still occurs on any genuinely low-salience span.

Backward compatibility

Additive. Old ledgers without om.observations.none markers behave exactly as
before. Unknown custom types are already ignored by fold/projection/view
(no exhaustive customType switch), so a downgrade after this lands degrades
gracefully (an om.observations.none entry is simply skipped, reverting to the
prior — buggy — behavior for that one span).

Testing

  • npm run typecheck — clean.
  • npm test181 passing (179 prior + 2 new).
  • Updated tests/consolidation-trigger.test.ts: the no-output case now asserts a
    single om.observations.none { coversUpToId } marker is appended (and no
    om.observations.recorded), reflector/dropper are not called, and the
    notification is info.
  • Added tests/session-ledger-progress.test.ts: a none marker advances
    observation coverage even with no recorded marker present, new source after it
    is still uncovered, and it does not advance reflection/drop coverage.
  • Added an observationsNoneEntry fixture + V3_OBSERVATIONS_NONE constant.

The observer fires whenever rawTokensSinceObservationCoverage >=
observeAfterTokens. On a non-empty result it appends an
om.observations.recorded marker, which advances the watermark. But when the
observer returns no observations, runObserverStage only notified and returned
"continue" without writing any coverage marker.

Because om.observations.recorded is the only thing that advanced observation
coverage (isValidCoverageEntry requires a non-empty observations array), a
single "nothing worth recording" verdict on a >= observeAfterTokens chunk left
the watermark stuck. The same span (growing by the new tail each turn) stayed
over the threshold, so the observer re-ran on every agent_start/turn_end,
re-emitting "observer returned no observations" and burning an observer model
call each time, until some later turn happened to yield an observation.

Fix: introduce a coverage-only marker om.observations.none { coversUpToId }.
On an empty observer result we append it; observation-coverage scans
(rawTokensSinceObservationCoverage and the observer chunk-start index) now
consider both om.observations.recorded and om.observations.none via
OBSERVATION_COVERAGE_TYPES. The marker carries no observations, so projection,
fold, reflector and dropper ignore it; reflection and drop watermarks are
unchanged (they still track their own marker types only). Raw entries remain
available to the reflector and until compaction, so no memory is lost.

Also downgrades the per-empty notification from "warning"
("observer returned no observations") to an "info"
("observer found nothing new to record"), since an empty verdict is now a
normal, one-shot event rather than a recurring error condition.

Tests: update the no-output consolidation test to assert the new coverage-only
marker contract; add progress-helper tests that a none marker advances
observation coverage without a recorded marker and does not touch
reflection/drop coverage. Full suite: 181 passing.
@elpapi42

elpapi42 commented Jul 3, 2026

Copy link
Copy Markdown
Owner

This is not a good idea, usually if the observer fails to extract any observations is because of technical errors, advancing the watermark will create a gap in the observed session. in the rare scenario the observer genuinely didn't have anything to observe, the next observer will have more meat/context to generate rich, useful observations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Observer re-fires every turn (warning spam) when it returns no observations

2 participants