Broker process not cleaned up on session exit — no idle timeout

[NewAiCoder]

Bug

The Codex companion app-server broker process (app-server-broker.mjs) persists indefinitely after the Claude Code session that spawned it exits. Found a stale broker (3 processes, ~185 MB) running 5+ hours after the originating session ended.

Root Cause

The plugin has a well-implemented session-lifecycle-hook.mjs with handleSessionEnd() that sends broker/shutdown — but this hook doesn't appear to fire reliably. The broker itself has no idle timeout — it only shuts down upon receiving an explicit broker/shutdown RPC or SIGTERM/SIGINT.

When the session-end hook fails to fire (or the session crashes/is killed), the broker runs forever.

Suggested Fix

Add an idle timeout to app-server-broker.mjs as a defensive fallback. If no RPC messages are received for N minutes (e.g., 15-30), the broker should self-terminate. This complements the existing session-end hook rather than replacing it.

The lifecycle hook approach is correct, but defense-in-depth with an idle timeout would prevent orphaned brokers when the hook doesn't fire.

Environment

• Ubuntu 22.04, Claude Code CLI
• @openai/codex-plugin-cc v1.0.2
• Broker was spawned for intake-portal project, outlived the session by 5+ hours

Workaround

Manually killing the broker processes after session exit.

[NewAiCoder]

Local fix applied

Added a 15-minute idle timeout to app-server-broker.mjs. The timer resets on every incoming socket data (any RPC activity) and calls the existing shutdown() for clean teardown.

+const IDLE_TIMEOUT_MS = Number(process.env.CODEX_BROKER_IDLE_TIMEOUT_MS) || 15 * 60 * 1000;
+let idleTimer = null;
+
+function resetIdleTimer(shutdownFn) {
+  if (idleTimer) clearTimeout(idleTimer);
+  idleTimer = setTimeout(async () => {
+    process.stderr.write(`Broker idle for ${IDLE_TIMEOUT_MS / 1000}s with no RPC activity, self-terminating.\n`);
+    await shutdownFn();
+    process.exit(0);
+  }, IDLE_TIMEOUT_MS);
+  idleTimer.unref();
+}

 // Before server creation:
+  resetIdleTimer(() => shutdown(server));

 // First line inside socket "data" handler:
+      resetIdleTimer(() => shutdown(server));

• Configurable via CODEX_BROKER_IDLE_TIMEOUT_MS env var
• .unref() ensures the timer doesn't prevent normal exit
• Uses existing shutdown() for clean teardown (sockets, pid file, unix socket)

Happy to submit a PR.

[Wintersta7e]

Hit this with v1.0.2 on Linux/WSL2.

Repro: prior session's codex:rescue skill call did not complete cleanly. Next session's first codex:rescue call hung indefinitely.

State at hang time:

• Detached app-server-broker.mjs serve process from a prior session still running (started ~30 min earlier, parent gone, PPID 1).
• Its Unix socket at /tmp/cxc-XXXXXX/broker.sock was accepting connections.
• ~/.claude/plugins/data/codex-openai-codex/state/<project-key>/broker.json recorded the orphan broker's PID + endpoint.
• Next session's loadBrokerSession returned the stale entry. isBrokerEndpointReady connected successfully (the 150 ms TCP-connect check passes). runAppServerTurn then hung at the first client.request(...) with no response.

Root cause seems narrower than "no idle timeout": even when the orphan broker is alive, it can be wedged at the application layer — accepts socket connections, doesn't answer JSON-RPC. scripts/lib/broker-lifecycle.mjs:isBrokerEndpointReady only opens a TCP connection, it never sends a JSON-RPC message. A connection-accepting-but-RPC-unresponsive broker passes the check, then the real call hangs forever.

I confirmed the wedged state by hand: echo '{}' | nc -U /tmp/cxc-XXXXXX/broker.sock connected fine but never got a response back — only timeout killed it.

The Codex CLI itself was healthy throughout — codex mcp-server (registered as a Claude Code MCP server) returned a session reply in <30 s with the same auth/config. So the wedge is plugin-side broker state, not Codex.

Workaround in client config — Claude Code SessionStart hook that kills orphan brokers and clears stale state before the plugin reuses it:

{
  "hooks": {
    "SessionStart": [{
      "matcher": "",
      "hooks": [{
        "type": "command",
        "command": "pkill -f 'app-server-broker.mjs serve' 2>/dev/null; find ~/.claude/plugins/data/codex-openai-codex/state -name broker.json -delete 2>/dev/null; exit 0"
      }]
    }]
  }
}

Tradeoff: kills brokers shared by concurrent sessions on the same host, but the next call respawns one cleanly. Acceptable for single-session users.

Suggested upstream fix (in addition to the idle-timeout proposal in this issue):

isBrokerEndpointReady should send a real JSON-RPC ping and require a valid response within a short timeout, not just succeed on socket-connect. Sketch:

async function isBrokerEndpointReady(endpoint, timeoutMs = 2000) {
  return new Promise((resolve) => {
    const socket = connectToEndpoint(endpoint);
    const deadline = setTimeout(() => { socket.destroy(); resolve(false); }, timeoutMs);
    socket.setEncoding("utf8");
    let buf = "";
    socket.on("connect", () => {
      socket.write(`${JSON.stringify({ id: 1, method: "broker/ping", params: {} })}\n`);
    });
    socket.on("data", (chunk) => {
      buf += chunk;
      const line = buf.split("\n")[0];
      try {
        const msg = JSON.parse(line);
        if (msg.id === 1) { clearTimeout(deadline); socket.end(); resolve(true); }
      } catch { /* keep buffering */ }
    });
    socket.on("error", () => { clearTimeout(deadline); resolve(false); });
  });
}

(Requires the broker to expose a cheap broker/ping method — broker/shutdown already exists, a ping should be straightforward to add alongside it.)

Combined with the idle timeout this issue proposes, the failure mode becomes self-recovering: stale broker.json on disk → ping times out → teardown + respawn happens automatically inside ensureBrokerSession.

[devseunggwan]

Hitting this on macOS with v1.0.4. Wanted to share an additional axis to the root cause that may complement the idle-timeout proposal.

Additional root cause: handleSessionEnd only handles a single cwd

In scripts/session-lifecycle-hook.mjs (v1.0.4):

async function handleSessionEnd(input) {
  const cwd = input.cwd || process.cwd();
  const brokerSession =
    loadBrokerSession(cwd) ??
    (process.env[BROKER_ENDPOINT_ENV] ? { ... } : null);
  ...
  teardownBrokerSession({ endpoint, pidFile, logFile, sessionDir, pid, ... });
}

loadBrokerSession(cwd) looks up the broker registered for the SessionEnd hook's cwd — i.e., a single workspace root. The teardown call below it then tears down that one broker.

In a Claude Code session where the agent issues /codex:review (or any companion call) against multiple worktrees over the session's lifetime, each call to ensureBrokerSession(cwd, …) in scripts/lib/app-server.mjs:340 spawns a fresh broker keyed to a different cwd. The state file at ~/.claude/plugins/data/codex-openai-codex/state/<workspace-key>/broker.json is per-workspace, so N reviews across N worktrees produce N independent broker triplets (broker.mjs + node wrapper + native codex app-server).

When SessionEnd eventually fires, only one of those N brokers is torn down. The remaining N-1 leak even when the hook fires reliably.

Observed in practice

After ~1 week of normal usage (multi-worktree workflow, agent-driven reviews across ~6 active worktrees, Claude Code session kept alive in a terminal multiplexer):

• 59 broker processes (app-server-broker.mjs serve …)
• 118 codex app-server processes (each broker spawns a node wrapper + native binary)
• Oldest broker: 7 days
• Memory: ~6 MB per process × 177 processes ≈ ~1 GB resident
• 59 corresponding stale /var/folders/.../T/cxc-*/ session directories with leftover broker.log files

SIGTERM to all 90 currently-active brokers + their children terminated cleanly without SIGKILL, and teardownBrokerSession's socket/pidfile unlink ran correctly per-broker (the cxc-* directories were left containing only a 0-byte broker.log afterwards). So the existing teardown function is sound — the gap is just that it's never invoked for non-current-cwd brokers.

Why the idle-timeout fix in this issue would resolve the multi-worktree variant too

The idle-timeout fallback proposed in the first comment self-terminates each broker after N minutes of socket inactivity, independent of which cwd the SessionEnd hook is looking at. So a single idle-timeout PR would absorb both:

1. The original "SessionEnd hook didn't fire" failure mode
2. This "SessionEnd hook fired but missed N-1 brokers" failure mode

Without idle-timeout, an alternative complementary fix would be to have handleSessionEnd enumerate all broker state files under the plugin's state directory (~/.claude/plugins/data/codex-openai-codex/state/*/broker.json) and tear down each, rather than only the one matching input.cwd. But idle-timeout is the simpler defense-in-depth choice — the per-cwd lookup can stay as the happy-path optimization.

Workaround (mass cleanup)

For others hitting this before the upstream fix lands, terminating all brokers older than N days is safe (existing shutdown() cleans up sockets + pidfiles correctly on SIGTERM):

# Kill all broker triplets older than 2 days
ps -axo pid,etime,command \
  | grep -E 'codex app-server|app-server-broker\.mjs' \
  | grep -v grep \
  | awk '{ if ($2 ~ /-/) { split($2, a, "-"); if (a[1]+0 >= 2) print $1 } }' \
  | xargs -n1 kill -TERM

# Clean up the leftover empty session directories
rm -rf "${TMPDIR}cxc-"*

Happy to file a separate issue for the per-cwd handler if it's worth tracking independently, but flagging here in case it changes the framing of the proposed fix.

[Pelton]

Reproducing this on macOS (Claude Code with codex@openai-codex v1.0.4): same symptom: stale brokers (app-server-broker.mjs plus node codex app-server plus Rust codex app-server) outlive SessionEnd, reparent to launchd (PPID 1), and accumulate across sessions until killed manually. Tracking PR #343 as the candidate cleanup fix.

[lfarkas]

Corroborating this with a much more severe real-world data point, plus a code-level confirmation of the root cause and a consumer-side workaround.

Severity is worse than "a stale broker for a few hours"

On a single dev box I found 112 orphaned processes across 14 dead sessions, the oldest 12 days old. Every leaked group is a full app-server-broker.mjs → codex app-server (node + rust) → MCP sidecars (context7-mcp, chrome-devtools-mcp) tree, each holding a /tmp/cxc-* socket dir. They accumulate one group per session that didn't exit gracefully, indefinitely.

The reliable signature of an orphan is exactly what #193 notes: app-server-broker.mjs with ppid==1 (reparented to init = its session is gone). A live session's broker still has a live parent, so this cleanly distinguishes leaked from active.

Code-level confirmation of the root cause

The broker's lifetime depends entirely on the SessionEnd hook firing and succeeding:

• lib/broker-lifecycle.mjs spawns it detached: true + child.unref(), so it is deliberately decoupled from the session process tree.
• app-server-broker.mjs only self-terminates on an explicit broker/shutdown RPC or SIGTERM/SIGINT (the process.on(...) handlers near the end). Its socket.on("close") handler only removes that one socket from the set — the last client disconnecting does not shut the broker down.
• Teardown is driven solely by session-lifecycle-hook.mjs → handleSessionEnd() → teardownBrokerSession().

So any non-graceful session exit — crash, kill -9, SSH/terminal hangup, OOM, or the hook simply not firing — leaves a detached, unref'd broker with no fallback path to ever die. This is why #108's idle-timeout suggestion is the right fix: the fallback has to live in the broker itself, because by definition the leak happens precisely when the hook-based teardown didn't run.

Worth combining a couple of the suggested fallbacks for full defense-in-depth:

• idle timeout (no RPC for N min → exit) — handles the genuinely-idle case (Broker process not cleaned up on session exit — no idle timeout #108)
• parent-liveness watchdog (poll the spawning PID; exit when it dies / ppid becomes 1) — handles crash/SIGKILL/hangup, where an idle timer alone would still leak for N minutes and Orphaned codex process at 100% CPU after Claude Code session ended (PPID=1, main thread spinning in read()) #193's busy-looping codex binary would never be idle at all

Workaround (consumer side)

Until the broker self-terminates, a Claude Code SessionStart hook that reaps any app-server-broker.mjs with ppid==1 (killing its whole subtree and removing the /tmp/cxc-* dir) cleans these up automatically at the start of each session. It's safe because the kill set is always rooted at an orphaned broker, so live sessions and unrelated processes are never touched.

Environment: @openai/codex-plugin-cc (codex plugin v1.0.4), codex-cli 0.142.2, Claude Code 2.1.193, Linux.