P1: Add concurrency governor for streaming and agent spawning

[galic1987]

Source

ChatGPT architecture review feedback

Problem

No global backpressure mechanism for streaming responses or multi-agent spawning. Unbounded task spawning with concurrent streaming can cause:

• Resource exhaustion under heavy swarm workloads
• Flaky behavior when many agents stream simultaneously
• No structured cancellation on session end

Proposal

1. Global semaphore (MAX_INFLIGHT_STREAMS) to cap concurrent LLM streaming responses
2. Per-agent semaphore to limit tool execution concurrency within each agent
3. Structured cancellation — when a session ends or ESC is pressed, all spawned tasks are cancelled cleanly via CancellationToken (tokio-util) rather than just setting an AtomicBool
4. Backpressure on tool results — if an agent's result queue is full, slow down rather than drop

Relevant Code

• src/agent/mod.rs — cancelled: Arc<AtomicBool> (current cancellation)
• src/agent/execution.rs — tool execution spawning
• src/orchestration/parallel.rs — parallel orchestration

Priority

P1 — stability under load

[m13v]

the pre-allocated semaphore approach is solid, we learned this the hard way running 3-5 Claude agents in parallel. dynamically adjusting concurrency limits mid-session caused race conditions constantly - one agent would start a streaming response while another was tearing down, and the shared resource counters would drift. fixed allocation at startup plus a strict claim/release pattern eliminated those bugs entirely.

for structured cancellation, CancellationToken is the right abstraction but watch out for tool executions that spawn their own subtasks. we had cases where cancelling the parent task would leave orphaned HTTP connections or file handles from tools that were mid-execution. adding a per-tool cleanup hook that runs on cancellation made things much cleaner.

on the backpressure question - slowing down is definitely better than dropping. we started with dropping and it caused cascading failures because the orchestrator assumed results that never arrived. a bounded channel with back-pressure propagation up to the LLM streaming layer worked well, the model just pauses its token output until there is room.

[m13v]

our parallel agent orchestration layer that handles the session lifecycle, cancellation, and resource isolation patterns: https://github.com/m13v/tmux-background-agents/blob/main/SKILL.md

and for the tool execution concurrency side, our function calling executor that manages parallel tool invocations from the LLM with proper cleanup: https://github.com/m13v/fazm/blob/main/Desktop/Sources/Providers/ChatToolExecutor.swift