[mara] prod incident: prod-down

[vaderyang]

🤖 mara detected a production heron incident.

• Signature: prod-down
• What: heron health endpoint unreachable or non-200 (HTTP 000)
• Health URL: http:///api/health
• HTTP: 000
• Observed (UTC): 2026-06-20T19:34:46Z
• Confirmed: failed 3/3 consecutive polls (~10s apart) — not a transient/deploy blip

Health response

000

Log context

---- tail ----
2026-06-02T02:09:28.487022242Z DEBUG h_storage::buffer: flushed entity="metrics" batch_len=38 trigger="shutdown" flush_ms=37
2026-06-02T02:09:28.487121246Z DEBUG h_storage::buffer: write buffer stopping: upstream EOF entity="metrics"
2026-06-02T02:09:31.332403253Z  INFO h_common::internal_metrics: [INTERNAL] global | turn     | agent_turns_open=0
2026-06-02T02:09:31.332519823Z  INFO h_common::internal_metrics: [INTERNAL] global | storage  | buf_calls=0 buf_turns=0 buf_metrics=0 buf_exchanges=0 flushed_calls=420/0 flushed_turns=12/0 flushed_metrics=1859/53 flushed_exchanges=852/4 flush_errors=0/0 q_calls=0/4096(0%) q_turns=0/4096(0%) q_metrics=0/4096(0%) q_exchanges=0/4096(0%)
2026-06-02T02:09:31.332498593Z  INFO h_common::internal_metrics: [INTERNAL] pipeline.local | capture  | pkts_received=35354/81 pkts_dropped_kernel=572/0 pkts_truncated=0/0 batches_received=0/0 batches_dropped_zmq=0/0 heartbeats_emitted=1482/3 read_errors=0/0 dump_errors=0/0 dump_late_minute_pkts=0/0 dump_retention_files_deleted=0/0 dump_retention_bytes_deleted=0/0 dump_retention_errors=0/0
2026-06-02T02:09:31.332577399Z  INFO h_common::internal_metrics: [INTERNAL] pipeline.local | protocol | pkts_routed=35354/81 flow_heartbeats_received=5928/12 flow_heartbeats_dropped=0/0 pkts_parsed=35354/81 pkts_dropped_not_ip=0/0 pkts_dropped_not_tcp=0/0 pkts_dropped_malformed=0/0 http_reqs_parsed=1298/8 http_resps_parsed=852/4 sse_events_parsed=5090/0 http_resyncs=446/4 tcp_ooo_dropped=29/0 tcp_ooo_buffered=283/0 tcp_rexmits_ignored=5362/4 flows_expired=563/1 flows_active=19 http_exchanges_joined=852/4 http_exchanges_unpaired=0/0 http_exchanges_expired=0/0 http_exchanges_pending=0 joiner_heartbeats_received=5928/12 q_raw_pkts=0/4096(0%) q_parsed_pkts=0/4096(0%)
2026-06-02T02:09:31.332597642Z  INFO h_common::internal_metrics: [INTERNAL] pipeline.local | llm      | wires_detected=435/0 wires_ignored=863/8 calls_with_agent=235/0 calls_without_agent=185/0 generic_tool_id_canonicalized=160/0 generic_session_id_synth_failed=0/0 llm_heartbeats_received=5928/12 tokens_estimated=11/0 q_http_parse_events=0/4096(0%) q_http_joiner_events=0/4096(0%) classifier_unknown=0/0 classifier_mixed=0/0
2026-06-02T02:09:31.332611974Z  INFO h_common::internal_metrics: [INTERNAL] pipeline.local | turn     | calls_ingested=235/0 calls_auxiliary=8/0 calls_dropped_late=1/0 turns_completed=12/0 turns_closed_grace=9/0 turns_closed_idle=3/0 turns_discarded_no_user_start=9/0 turn_heartbeats_received=5928/12 turn_heartbeats_dropped=0/0 turn_calls_buffered=0 q_agent_calls=0/4096(0%)
2026-06-02T02:09:31.332623887Z  INFO h_common::internal_metrics: [INTERNAL] pipeline.local | metrics  | llm_events_start=435/0 llm_events_complete=420/0 metrics_heartbeats_received=5928/12 metrics_heartbeats_dropped=0/0 windows_emitted=1859/53 aggregator_open_buckets=0 aggregator_concurrency_table=27 q_llm_events=0/4096(0%)
2026-06-02T02:09:33.331593535Z ERROR heron: pipeline drain timed out; forcing shutdown timeout_secs=5
2026-06-02T02:09:33.332064503Z  INFO h_common::internal_metrics: [INTERNAL] pipeline.local | capture  | pkts_received=35354/0 pkts_dropped_kernel=572/0 pkts_truncated=0/0 batches_received=0/0 batches_dropped_zmq=0/0 heartbeats_emitted=1482/0 read_errors=0/0 dump_errors=0/0 dump_late_minute_pkts=0/0 dump_retention_files_deleted=0/0 dump_retention_bytes_deleted=0/0 dump_retention_errors=0/0
2026-06-02T02:09:33.332124091Z  INFO h_common::internal_metrics: [INTERNAL] pipeline.local | protocol | pkts_routed=35354/0 flow_heartbeats_received=5928/0 flow_heartbeats_dropped=0/0 pkts_parsed=35354/0 pkts_dropped_not_ip=0/0 pkts_dropped_not_tcp=0/0 pkts_dropped_malformed=0/0 http_reqs_parsed=1298/0 http_resps_parsed=852/0 sse_events_parsed=5090/0 http_resyncs=446/0 tcp_ooo_dropped=29/0 tcp_ooo_buffered=283/0 tcp_rexmits_ignored=5362/0 flows_expired=563/0 flows_active=19 http_exchanges_joined=852/0 http_exchanges_unpaired=0/0 http_exchanges_expired=0/0 http_exchanges_pending=0 joiner_heartbeats_received=5928/0 q_raw_pkts=0/4096(0%) q_parsed_pkts=0/4096(0%)
2026-06-02T02:09:33.332143075Z  INFO h_common::internal_metrics: [INTERNAL] pipeline.local | llm      | wires_detected=435/0 wires_ignored=863/0 calls_with_agent=235/0 calls_without_agent=185/0 generic_tool_id_canonicalized=160/0 generic_session_id_synth_failed=0/0 llm_heartbeats_received=5928/0 tokens_estimated=11/0 q_http_parse_events=0/4096(0%) q_http_joiner_events=0/4096(0%) classifier_unknown=0/0 classifier_mixed=0/0
2026-06-02T02:09:33.332187385Z  INFO h_common::internal_metrics: [INTERNAL] pipeline.local | turn     | calls_ingested=235/0 calls_auxiliary=8/0 calls_dropped_late=1/0 turns_completed=12/0 turns_closed_grace=9/0 turns_closed_idle=3/0 turns_discarded_no_user_start=9/0 turn_heartbeats_received=5928/0 turn_heartbeats_dropped=0/0 turn_calls_buffered=0 q_agent_calls=0/4096(0%)
2026-06-02T02:09:33.332199567Z  INFO h_common::internal_metrics: [INTERNAL] pipeline.local | metrics  | llm_events_start=435/0 llm_events_complete=420/0 metrics_heartbeats_received=5928/0 metrics_heartbeats_dropped=0/0 windows_emitted=1859/0 aggregator_open_buckets=0 aggregator_concurrency_table=27 q_llm_events=0/4096(0%)
2026-06-02T02:09:33.332216067Z  INFO h_common::internal_metrics: [INTERNAL] pipeline.local | metrics reporter stopped
2026-06-02T02:09:33.332649578Z  INFO h_common::internal_metrics: [INTERNAL] global | turn     | agent_turns_open=0
2026-06-02T02:09:33.332694153Z  INFO h_common::internal_metrics: [INTERNAL] global | storage  | buf_calls=0 buf_turns=0 buf_metrics=0 buf_exchanges=0 flushed_calls=420/0 flushed_turns=12/0 flushed_metrics=1859/0 flushed_exchanges=852/0 flush_errors=0/0 q_calls=0/4096(0%) q_turns=0/4096(0%) q_metrics=0/4096(0%) q_exchanges=0/4096(0%)
2026-06-02T02:09:33.332708435Z  INFO h_common::internal_metrics: [INTERNAL] global | metrics reporter stopped
2026-06-02T02:09:33.33335498Z ERROR heron: graceful shutdown stalled; exiting forcefully

---

Filed automatically by mara (prod observer). Dedup window: 6h.
Add agent:assess to route this to the triage → wiwi loop once confirmed actionable.

[vaderyang]

ebpf fd leak. fixed.