Skip to content

Ingestion should auto-stop (or divert) when inference is hard-down beyond a configurable interval #90

Description

@antra-tess

Problem

When an agent cannot complete any inference (e.g. `OverBudgetError` / adaptive-picker exhaustion, persistent provider failure), the host keeps ingesting channel events into the store indefinitely. Two compounding harms:

  1. The failure deepens itself. In the Lena incident (2026-08-04 → 08-06), the store was 2,138 tokens over hard budget when compile first failed. Ingestion continued for two days while she was mute: 13,485 → 16,464 chunks, middle 564k → 865k (+135k/day). A marginal, easily-recoverable overrun grew into a 270k structural deficit — every hour down made recovery strictly harder.

  2. Time-vortex on recovery. Whatever eventually unwedges the agent confronts it with thousands of unseen stale messages spanning days, which it experiences as a wall of "now" (or must be branched away by an operator, as we did here — `prewedge-20260806`, 3,211 messages taken off the active timeline).

Proposal

If inference has been failing continuously for longer than a configurable interval (suggest default 1h, `ingestionStopAfterInferenceDownMs` or similar), the host stops appending ingested channel events to the agent's timeline:

  • Events should be diverted or parked (e.g. buffered off-timeline, or simply dropped with a counter) rather than appended — the operator can decide on recovery what, if anything, to replay.
  • Wake-attempts can continue (each failure re-checks whether inference recovered), but the store must stop growing.
  • On recovery, inject a single seam marker ("ingestion was suspended HH:MM–HH:MM, N events not recorded") so the gap is legible to the resident rather than silent — same doctrine as announced surgery seams.
  • The existing `[inference-hard-down]` consecutive-failure tracking is probably the right trigger signal; interval-based (wall-clock since first failure of the streak) rather than count-based, since failure cadence varies with traffic.

Incident receipts (Lena, opus-4-6, Mac residence)

  • First failure: 2026-08-04T11:13:57Z `Adaptive picker exhausted but 597426 tokens still exceed hard budget 595288` (2k over).
  • 140 consecutive failures over ~46h; final: `865330 tokens ... 16464 chunks, deepest fold level=L4`.
  • Root cause of the wedge itself was a compression-merge failure mode (separate issue/fix in context-manager: merge mints answered with an empty `think` tool call — `unusable_empty, stop=tool_use` ×97); but the unbounded growth while down is a host-level failure amplifier independent of what wedged inference.

Requested by antra 2026-08-06.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions