feat(runtime): pool node/python runtime processes across agents and skill runs (#5106)

.env.example

@@ -336,6 +336,25 @@ OPENHUMAN_SKILLS_WORKING_MEMORY_ENABLED=true
 # [optional] Reserved cache directory for future managed CPython installs.
 # OPENHUMAN_RUNTIME_PYTHON_CACHE_DIR=
 
+# ---------------------------------------------------------------------------
+# Shared language-runtime pool (#5106)
+# ---------------------------------------------------------------------------
+# Reuse a small, bounded set of warm node/python worker processes across skill
+# runs and node_exec instead of forking one interpreter child per run. Full
+# tuning lives in config.toml `[runtime_pool]`; these are the operator knobs.
+# Node pooling is ON by default (each job runs in an isolated worker_thread).
+# Python pooling is OFF by default (jobs share one interpreter, so reuse can
+# leak globals across runs); opt in with `[runtime_pool.python] enabled = true`.
+# [optional] Master kill switch. Default: true. `false` reverts every caller to
+# the legacy per-call spawn (no behavioural change, just no reuse).
+# OPENHUMAN_RUNTIME_POOL_ENABLED=true
+# [optional] Max concurrently-resident node workers (default 2). Work beyond
+# this queues rather than forking a new interpreter.
+# OPENHUMAN_RUNTIME_POOL_NODE_MAX_WORKERS=2
+# [optional] Max concurrently-resident python workers (default 2). Only takes
+# effect when python pooling is explicitly enabled (see above).
+# OPENHUMAN_RUNTIME_POOL_PYTHON_MAX_WORKERS=2
+
 # ---------------------------------------------------------------------------
 # TokenJuice — content-aware tool-output compaction (the content router)
 # ---------------------------------------------------------------------------

docs/library-benchmarking.md

@@ -38,7 +38,7 @@ stderr). Each models a distinct embedding use case:
 
 ## How to run
 
-Five scripts under `scripts/profile/` (each has `-h`/`--help`):
+Six scripts under `scripts/profile/` (each has `-h`/`--help`):
 
 - **`library-bench.sh`** — the primary RSS/duration benchmark. Builds the
   binaries, runs each scenario N fresh-process repeats (default 5), and
@@ -88,6 +88,14 @@ Five scripts under `scripts/profile/` (each has `-h`/`--help`):
   ./scripts/profile/library-instances.sh --instances "10,25,50" --hold-secs 30
   ```
 
+- **`library-pool-gate.sh`** — the runtime-pool regression gate (#5106). Runs
+  `skill-run` with K parallel skill runs and asserts the process tree grows by
+  ~one pooled worker, not K interpreters; reports pooled vs unpooled.
+
+  ```bash
+  ./scripts/profile/library-pool-gate.sh --concurrency 8 --workers 1
+  ```
+
 ### Default vs slim builds
 
 Default-feature builds link every compile-time domain gate (`voice`, `web3`,
@@ -115,6 +123,9 @@ behavior, not linked code size.
 | `OPENHUMAN_PROFILE_FORCE_UTC=1` | Skip `iana_time_zone`/CoreFoundation timezone resolution. |
 | `OPENHUMAN_PROFILE_HOLD_SECS` / `HOLD_BEFORE_SECS` | Pause the process at settled/baseline state for external inspection (`vmmap`, `heap`, `malloc_history`, Instruments). |
 | `OPENHUMAN_PROFILE_DHAT_OUT` | Output path for dhat JSON (set by `library-heap.sh`). |
+| `OPENHUMAN_PROFILE_SKILL_RUN_CONCURRENCY` | `skill-run`: number of parallel `code_executor` turns (K), each spawning a `node_exec` job (default 1). |
+| `OPENHUMAN_PROFILE_SKILL_RUN_POOL` | `skill-run`: `off` disables the shared runtime pool (legacy per-call spawn; tree then shows ~K resident `node` children). Default on. |
+| `OPENHUMAN_PROFILE_SKILL_RUN_POOL_WORKERS` | `skill-run`: pool size W when pooling is on (default 1). The scenario asserts `child_count <= W` for K > 1 — the #5106 regression gate. |
 
 ## Metrics and interpretation
 
@@ -334,6 +345,28 @@ Runtime/subagent scenarios: `skill-run` measures a real `node` child at
 runtime-pooling issue (tinyhumansai/openhuman#5106); `subagent-storm` shows
 ~0.78 MiB marginal per additional parallel subagent (K=8→32 cross-width).
 
+Runtime pool (#5106): with the shared pool on (the default), a batch of K
+concurrent `skill-run` turns shares a bounded set of warm `node` workers, so
+the process tree grows by ~one pooled worker instead of K interpreters.
+Measured at **K=8** (`max_workers=1`, system node v24): pooled tree
+`child_count=1` at **~202 MiB**, vs unpooled `child_count=8` at **~690 MiB**
+(eight `node` children of ~73–75 MB each) — a ~490 MiB / 3.4× reduction that
+grows with K. Compare the two regimes directly:
+
+```bash
+# Legacy: K interpreters resident at peak.
+OPENHUMAN_PROFILE_SKILL_RUN_CONCURRENCY=8 OPENHUMAN_PROFILE_SKILL_RUN_POOL=off \
+  target/release/library-profile skill-run
+# Pooled: child_count stays at the pool size (asserted), not K.
+OPENHUMAN_PROFILE_SKILL_RUN_CONCURRENCY=8 OPENHUMAN_PROFILE_SKILL_RUN_POOL_WORKERS=1 \
+  target/release/library-profile skill-run
+```
+
+The pool is configured in `[runtime_pool]` (master switch + per-language
+`node`/`python` `max_workers`, `idle_ttl_secs`, `recycle_after_jobs`,
+`max_queue_depth`); `enabled = false` reverts every caller to the legacy
+per-call spawn.
+
 Watch-items from the sweep: thread count grows ~0.35/agent (needs
 attribution + cap before real 1000-agent runs), and p95 latency at N=500 on
 2 workers shows CPU saturation is the load constraint, not memory.

gitbooks/features/native-tools/coder.md

@@ -39,5 +39,5 @@ Filesystem tools respect a workspace boundary - the agent can't read or write ou
 
 ## See also
 
-* [System & Utilities](system-and-utilities.md) - `shell`, `node_exec`, `npm_exec` for the rest of the dev loop.
+* [System & Utilities](system-and-utilities.md) - `shell`, `node_exec`, `npm_exec`, `python_exec` for the rest of the dev loop.
 * [Agent Coordination](agent-coordination.md) - `todo_write`, `spawn_subagent` for larger refactors.

gitbooks/features/native-tools/system-and-utilities.md

@@ -14,6 +14,7 @@ The catch-all family. Small, sharp tools the agent reaches for to round out a ta
 | `shell`             | Run a shell command. Bounded output, captured exit code.                      |
 | `node_exec`         | Run a Node.js snippet - useful for one-off scripting.                         |
 | `npm_exec`          | Run an `npm`/`pnpm`/`yarn` script.                                            |
+| `python_exec`       | Run a Python 3 snippet or `.py` script - one-off scripting in Python.         |
 | `current_time`      | Get the current time in any timezone, with formatting options.                |
 | `schedule`          | One-shot "do this once at time T" - for recurring jobs see [Cron](cron.md).   |
 | `pushover`          | Send a push notification to your devices.                                     |

scripts/profile/README.md

@@ -94,6 +94,23 @@ exit or missing/invalid JSON); default is report-only. See
 [`docs/library-benchmarking.md`](../../docs/library-benchmarking.md#fleet-one-process-vs-instances-many-processes)
 for the fleet-vs-instances framing.
 
+### `library-pool-gate.sh` — runtime pool regression gate (#5106)
+
+Drives the `skill-run` scenario with K parallel skill runs and asserts the DoD:
+with the shared runtime pool ON, the process tree grows by ~one pooled worker,
+**not** K interpreters. The scenario hard-asserts `tree.child_count <= max_workers`
+for K > 1 (nonzero exit); this script runs it and reports a pooled-vs-unpooled
+comparison. A regression that reintroduces per-run forking fails the gate.
+
+```bash
+./scripts/profile/library-pool-gate.sh                     # K=8, max_workers=1
+./scripts/profile/library-pool-gate.sh --concurrency 16 --skip-build
+```
+
+Exits 0 = pass, 1 = regression. SKIPs (exit 0) when no system `node` is on
+`PATH` (the scenario must never download an interpreter), so node-less CI
+runners don't false-fail.
+
 ## Quick start
 
 ```bash

scripts/profile/library-pool-gate.sh

@@ -0,0 +1,133 @@
+#!/usr/bin/env bash
+# library-pool-gate.sh — regression gate for the shared runtime pool (#5106).
+#
+# Runs the `skill-run` scenario with K parallel skill runs and asserts the DoD:
+# with the pool ON, the process tree grows by ~one pooled worker, NOT K
+# interpreters. The scenario itself hard-asserts `tree.child_count <= max_workers`
+# for K > 1 (nonzero exit on failure); this script drives it in the profiling
+# suite and adds a pooled-vs-unpooled comparison for the report.
+#
+# A regression that reintroduces per-run interpreter forking makes the pooled
+# run's child_count scale with K → the scenario exits nonzero → this gate fails.
+#
+# Usage:
+#   ./scripts/profile/library-pool-gate.sh [--concurrency N] [--workers W] [--skip-build] [--out DIR]
+#
+# Exits 0 = pass, 1 = regression/failure, 0 (with SKIP notice) = no system node.
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+
+CONCURRENCY=8
+POOL_WORKERS=1
+SKIP_BUILD=0
+OUT_DIR=""
+
+# Reject non-positive-integer values up front so a bad --concurrency/--workers
+# fails loudly instead of silently changing the workload.
+require_pos_int() {
+    case "$2" in
+        ''|*[!0-9]*) echo "ERROR: $1 must be a positive integer, got: $2" >&2; exit 1 ;;
+    esac
+    [ "$2" -ge 1 ] || { echo "ERROR: $1 must be >= 1, got: $2" >&2; exit 1; }
+}
+
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --concurrency) CONCURRENCY="${2:?--concurrency requires a value}"; require_pos_int --concurrency "$CONCURRENCY"; shift 2 ;;
+        --workers) POOL_WORKERS="${2:?--workers requires a value}"; require_pos_int --workers "$POOL_WORKERS"; shift 2 ;;
+        --skip-build) SKIP_BUILD=1; shift ;;
+        --out) OUT_DIR="${2:?--out requires a value}"; shift 2 ;;
+        -h|--help) sed -n '2,18p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;;
+        *) echo "ERROR: unknown argument: $1" >&2; exit 1 ;;
+    esac
+done
+require_pos_int --concurrency "$CONCURRENCY"
+require_pos_int --workers "$POOL_WORKERS"
+
+log() { echo "[pool-gate] $*" >&2; }
+
+if ! command -v jq >/dev/null 2>&1; then
+    echo "ERROR: jq is required. Install it (e.g. 'brew install jq')." >&2
+    exit 1
+fi
+
+# The scenario requires a real system node (it must never download one). No node
+# ⇒ SKIP (exit 0) so node-less CI runners don't false-fail; a node-equipped
+# runner is where this gate is meaningful.
+if ! command -v node >/dev/null 2>&1; then
+    log "SKIP: no system 'node' on PATH — pool gate needs a real interpreter."
+    exit 0
+fi
+
+[[ -z "$OUT_DIR" ]] && OUT_DIR="$REPO_ROOT/target/profile/rust-library/pool-gate-$(date +%Y%m%d-%H%M%S)"
+mkdir -p "$OUT_DIR"
+BIN="$REPO_ROOT/target/release/library-profile"
+
+if [[ "$SKIP_BUILD" -eq 0 ]]; then
+    log "building library-profile (rss-bench, GGML_NATIVE=OFF)"
+    ( cd "$REPO_ROOT" && GGML_NATIVE=OFF cargo build --release --features rss-bench --bin library-profile )
+fi
+[[ -x "$BIN" ]] || { echo "ERROR: $BIN not found (drop --skip-build to build it)." >&2; exit 1; }
+
+pooled_json="$OUT_DIR/pooled-k$CONCURRENCY.json"
+unpooled_json="$OUT_DIR/unpooled-k$CONCURRENCY.json"
+
+# --- Pooled run (the gate) ------------------------------------------------
+# The scenario hard-asserts child_count <= POOL_WORKERS; a nonzero exit here
+# (via set -e) fails the gate.
+log "pooled run: K=$CONCURRENCY, max_workers=$POOL_WORKERS (asserts child_count <= $POOL_WORKERS)"
+# Force the pool ON explicitly: an inherited OPENHUMAN_PROFILE_SKILL_RUN_POOL=off
+# would run the legacy path and make the scenario skip its pool assertion.
+OPENHUMAN_PROFILE_SKILL_RUN_POOL=on \
+OPENHUMAN_PROFILE_SKILL_RUN_CONCURRENCY="$CONCURRENCY" \
+OPENHUMAN_PROFILE_SKILL_RUN_POOL_WORKERS="$POOL_WORKERS" \
+    "$BIN" skill-run >"$pooled_json"
+
+# --- Unpooled baseline (report only) --------------------------------------
+log "unpooled baseline: K=$CONCURRENCY, pool OFF (expect ~$CONCURRENCY interpreters)"
+OPENHUMAN_PROFILE_SKILL_RUN_CONCURRENCY="$CONCURRENCY" \
+OPENHUMAN_PROFILE_SKILL_RUN_POOL=off \
+    "$BIN" skill-run >"$unpooled_json"
+
+# Pooled run: a missing/null tree means nothing was measured — that must FAIL,
+# not silently coerce to 0 (which would let the gate pass without observing an
+# interpreter). Unpooled is report-only, so its tree may default to 0.
+pooled_cc="$(jq -r '.tree.child_count // "null"' "$pooled_json")"
+unpooled_cc="$(jq -r '.tree.child_count // 0' "$unpooled_json")"
+pooled_rss="$(jq -r '.tree.tree_rss_kib // "null"' "$pooled_json")"
+unpooled_rss="$(jq -r '.tree.tree_rss_kib // 0' "$unpooled_json")"
+
+echo
+echo "=== runtime pool gate (#5106) — K=$CONCURRENCY ==="
+echo "  pooled   (max_workers=$POOL_WORKERS): child_count=$pooled_cc  tree_rss_kib=$pooled_rss"
+echo "  unpooled (legacy spawn)            : child_count=$unpooled_cc  tree_rss_kib=$unpooled_rss"
+
+if [[ "$pooled_cc" == "null" ]]; then
+    echo "FAIL: pooled run captured no process-tree sample — cannot verify the pooled worker." >&2
+    exit 1
+fi
+# The pooled worker must actually be observed (>= 1) and must not exceed the
+# pool size.
+if [[ "$pooled_cc" -lt 1 ]]; then
+    echo "FAIL: pooled run observed zero interpreter children — the pooled worker was not sampled." >&2
+    exit 1
+fi
+if [[ "$pooled_cc" -gt "$POOL_WORKERS" ]]; then
+    echo "FAIL: pooled child_count=$pooled_cc exceeds max_workers=$POOL_WORKERS — pool is forking per run." >&2
+    exit 1
+fi
+
+# Sanity: the unpooled baseline should fork more than the pool. If it didn't,
+# the comparison is inconclusive (likely a sampling miss on a fast machine) —
+# warn rather than hard-fail, since the pooled assertion above is the real gate.
+if [[ "$unpooled_cc" -le "$pooled_cc" ]]; then
+    echo "WARN: unpooled baseline child_count=$unpooled_cc did not exceed pooled=$pooled_cc" >&2
+    echo "      (baseline sampling inconclusive; the pooled assertion still passed)." >&2
+else
+    echo "PASS: pool bounded interpreters to $pooled_cc for K=$CONCURRENCY concurrent skill runs (unpooled forked $unpooled_cc)."
+fi
+echo "  artifacts: $OUT_DIR"
+exit 0