Field report: a week of unattended overnight runs — what held, and three gaps

[Polloinfilzato]

What this is

I ran bmad-loop 0.8.1 unattended overnight, most nights, for about a week on a real project
(~30 stories across several epics, scm.isolation = "worktree", claude adapter, macOS). The goal
was the obvious one: start a run at night, wake up to it done.

To make that survivable I built a supervision layer around the engine — a shell poller plus a
triage protocol, never anything inside the control loop. This is a report on what that exercise
found. I'm not asking you to adopt any of it. Three of the findings are already filed as
separate issues; the rest is offered as operational data.

The headline, including the part that embarrasses me

bmad-loop is well defended inside a run. Nothing defends the layer above it.

I built guards for twelve failure modes. On finally reading the engine source properly, three
of mine turned out to duplicate defences you already had — and yours were better:

• Host suspend freezing the session timeout. I added a wall-clock stall backstop. You already
  co-bound spec.timeout_s with a wall clock (session-end journaled 2h19 after session_timeout_min fires when the session is wedged inside a tool call #157) — and your version is more careful than mine
  would have been, because it lets the wall clock only expire a deadline, never extend it,
  guarding a backward clock step I hadn't considered.
• A long dev turn looking like a stall. I hashed the tmux pane to re-arm my stall clock. Your
  grace window already re-arms on pane-log activity.
• A session dying without a Stop event. I polled liveness from outside. You already score a
  dead window as crashed.

I'm reporting these because they were wrong. A supervisor watching from outside sees symptoms
and infers the mechanism is missing — it usually isn't. That's worth knowing about any external
tooling built against this project, including mine.

What survived that filter is narrow and, I think, real.

The gap that survived: nothing acts above the run

Inside a run you handle crashes, stalls, suspends, timeouts, and you preserve failed work with
keep_failed. But once the engine pauses — or the engine process itself dies — nothing acts.
There's no watchdog and no auto-resume, by design as far as I can tell. gates.notify fires and
then it's on a human.

For an attended run that's exactly right. For the overnight case it means one mechanical hiccup
at 03:00 costs the entire window — which is the whole economic argument for an unattended
orchestrator. Every guard I built that turned out to be necessary lives at that level.

Two concrete consequences, both filed separately:

• notify.desktop = true silently does nothing on macOS and Windows #231 — notify.desktop is Linux-only, so on macOS/Windows the one channel that could reach
  a sleeping human is silently inert. I hit this on a mount-failure deferral: the engine did
  everything right, journalled and notified, and I found out the next morning.
• A worktree_seed directory entry is skipped entirely when any child is tracked #230 — a worktree_seed directory entry is skipped whole when any child is tracked, with
  no diagnostic.

And one I haven't filed yet because it's a design change rather than a defect:

There's no non-destructive way past an escalation under worktree isolation. Phase.ESCALATED
is terminal so resume skips the story; resolve re-drives from scratch; --restore-patch is
rejected under worktree isolation. The work is kept (keep_failed leaves the branch mounted)
— it just can't be continued.

That matters because in my experience most escalations were false positives. The recurring
one: the review session's VC-hygiene gate raises a CRITICAL escalation ("working tree is dirty…")
because the dev pass left artifact files uncommitted — while the same message self-reports
Status: done with a full delivery summary. Mechanical block, not a quality objection. Three
separate stories.

So every recovery I do is a hand-rolled version of one missing command — merge the kept branch,
mark the story done, start a fresh run. Used four times, zero work lost; one of them avoided a
~28M-token re-drive. #172/#173 established the same principle for the dev-retry path; this is its
sibling on the escalation path. Happy to open it as a proper feature request if it's welcome —
CONTRIBUTING.md points at Discord for changes this size, so I haven't just filed it.

On "no LLM in the control loop"

Worth pre-empting, since my layer involves a model. The control loop stays yours: deterministic
Python, no model. Mine sits outside it and only wakes once the deterministic loop has already
stopped and is waiting for a human. What decides when to wake it is a shell script that does no
reasoning — it polls read-only state and, on the first actionable condition, writes a JSON event
and exits. The exit is the wake signal; a process that exits can't fail silently, which
matters because the failure that started all this was an API error that killed a dev turn without
firing the Stop hook at all. It doesn't put a model in the loop; it replaces the human who
would otherwise have to be awake.

Two things worth stealing, neither of them code

Autonomy by class of problem, not by confidence. I ended up with four tiers: mechanical and
known → fix and resume; verified false positive → finalize and continue; real doubt → get a second
model's opinion; no safe path or irreversible risk → stop and wake me. Only the last one wakes a
human, and the boundary is what kind of problem it is, not how sure the supervisor feels.

Finalizing a story without its review creates review debt, and a log line is not a queue. One
story I finalized as a false positive was correct to finalize — but it skipped that story's
automated review, and it then sat marked done for two days like any reviewed story. When the
review finally ran it found a HIGH-severity security defect. The obligation has to land somewhere
that blocks. Worth noting the story most likely to be finalized that way is a security-sensitive
one, because those are the ones already flagged for follow-up.

What I'd offer, if any of it is useful

In order of what I think is worth your time: the escalation path (above); then the two filed
issues; then, only if someone asks, the supervisor itself as a reference implementation. It's
~1,700 lines of shell and a protocol document, three components are macOS-only with one-for-one
Linux equivalents, and two scripts assume my project's sprint layout. Nobody adopts another
project's bash — the transferable parts are the tier ladder and the exit-to-wake shape.

One honest caveat on scope: everything above is from sequential runs. I also built parallel
supervision by starting concurrent --story runs, and that half is reasoned from source and
unit-tested but has never met a real merge-back collision. I wouldn't want the sequential
evidence to lend it credibility it hasn't earned. (The one field datum from that half is already
in #229.)

Happy to expand on any of this, or to leave it as a data point.

[coderabbitai]

🔗 Related PRs

#77 - feat(stories): spec-to-loop engine, adapter, CLI + HITL checkpoints (Phase 2 + 3) [merged]
#86 - feat(resolve): intent-gap patch-restore re-drive + inline-review-layer preflight (re-cut) [merged]
#95 - feat(adapters): post-kill reconcile rescues finished stalled/timeout sessions [merged]
#182 - fix(worktree): degrade post-merge teardown failures instead of crashing the run (#139) [merged]
#219 - feat: graceful stop for bmad-loop runs (finish the in-flight item, then stop — resumable) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[Haven2026]

Additional field data (different project, same 0.8.1 shape)

We hit the same “well defended inside a run; nothing defends the layer above” headline on a multi-epic unattended campaign (worktree, claude dev + codex review, sprint-status).

Two concrete engine gaps we filed separately (not asking you to adopt our outer salvage layer):

1. feat: optional remote Integration Closeout after local unit-merged (done ≠ origin) #270 — optional remote Integration Closeout: unit-merged / story-done is local only; projects whose real gate is origin/main need either in-engine remote closeout or a permanent external salvage.
2. [BUG] Review session timeout always RETRY until max_review_cycles; no salvage/converge exit #271 — review session timeout always RETRY until max_review_cycles; no salvage-if-done / timeout-class exit, so endgame stories thrash review-2/3 while product sits only in a worktree.

Operationally we mirrored your “autonomy by class of problem” idea outside the loop (stop thrash → product-only PR → board cite → never redispatch false greens). Happy to expand if useful; otherwise these two issues stand alone.