Skip to content

app-server brokers never self-terminate → orphaned process/RAM leak (34 chains, 272 procs, ~2.2GB) #543

Description

@cooperbi159

Bug: app-server brokers never self-terminate → orphaned process/RAM leak

Repo: openai/codex-plugin-cc
Version: codex plugin 1.0.6
File: scripts/app-server-broker.mjs (+ context in scripts/lib/broker-lifecycle.mjs, scripts/session-lifecycle-hook.mjs)

Summary

The shared app-server broker has no idle timeout. It exits only on an explicit
broker/shutdown RPC, SIGTERM, or SIGINT. Its only automatic reaper is the
SessionEnd hook, which is fragile. When that hook fails to run, the detached
broker (child.unref()) leaks forever, holding a Codex app-server child plus
its MCP/worker subprocesses.

Observed in the wild: 34 orphaned broker chains, 272 processes, ~2.2 GB RSS,
0 clients connected.
16 chains pointed at already-deleted git worktrees.

Root cause

Two gaps compound:

  1. Broker never self-exits. In app-server-broker.mjs, socket.on("close")
    just does sockets.delete(socket) + clearSocketOwnership(socket). There is
    no check for "no clients left" and no idle timer. A broker with zero clients
    lives indefinitely.

  2. The reaper is best-effort and misses common cases. SessionEnd in
    session-lifecycle-hook.mjs is the intended reaper, keyed per-cwd via
    loadBrokerSession(cwd). It fails to reap when:

    • Claude is hard-killed / crashes → hook never runs.
    • The 5s hook timeout (hooks.json) is exceeded on a busy machine → hook killed.
    • The worktree is deleted before SessionEndloadBrokerSession(cwd)
      resolves to gone state, returns null, no shutdown is sent → orphan.
      (This matches the 16 deleted-worktree chains.)

Because brokers are spawned detached and keyed to a cwd, any hook miss orphans a
broker that nothing will ever clean up.

Fix

Give the broker an idle self-exit. It is safe: the companion opens one socket
per CodexAppServerClient.connect() and closes it when the job ends
(BrokerCodexAppServerClient.initialize/close in lib/app-server.mjs), so a
broker at zero sockets with no active request/stream is genuinely idle. Killing
an idle broker only costs a cold respawn on the next job — ensureBrokerSession
already checks endpoint readiness and respawns if dead. This makes the broker
self-healing regardless of whether any hook fires, covering hard-kills and
deleted worktrees alike.

Threshold defaults to 15 min, overridable via CODEX_COMPANION_BROKER_IDLE_MS.

Patch (scripts/app-server-broker.mjs)

Add idle state after const sockets = new Set();:

  const IDLE_TIMEOUT_MS = Number(process.env.CODEX_COMPANION_BROKER_IDLE_MS ?? 15 * 60 * 1000);
  let idleTimer = null;
  let serverRef = null;

  function isIdle() {
    return sockets.size === 0 && !activeRequestSocket && !activeStreamSocket;
  }

  function clearIdleTimer() {
    if (idleTimer) {
      clearTimeout(idleTimer);
      idleTimer = null;
    }
  }

  function armIdleTimer() {
    clearIdleTimer();
    if (!(IDLE_TIMEOUT_MS > 0) || !serverRef || !isIdle()) {
      return;
    }
    idleTimer = setTimeout(async () => {
      if (!isIdle()) {
        return;
      }
      await shutdown(serverRef);
      process.exit(0);
    }, IDLE_TIMEOUT_MS);
    if (typeof idleTimer.unref === "function") {
      idleTimer.unref();
    }
  }

Cancel the timer on connect (in net.createServer callback, after sockets.add(socket)):

    clearIdleTimer();

Re-arm on disconnect (in both socket.on("close") and socket.on("error"),
after clearSocketOwnership(socket)):

      armIdleTimer();

Arm at startup and hold the server ref (replace server.listen(listenTarget.path);):

  serverRef = server;
  server.listen(listenTarget.path, () => {
    armIdleTimer();
  });

Optional hardening (not required if idle-exit lands)

  • Make the SessionEnd hook timeout longer than 5s, or move broker teardown off
    the hook's critical path, so slow machines still reap on clean exit.
  • On SessionStart, sweep os.tmpdir() for stale cxc-*/broker.pid whose pid is
    dead (leftover session dirs). Redundant once idle-exit ships, but cleans state.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions