From 1be74aa11e309103b93bc1183fcf06b4e032bf25 Mon Sep 17 00:00:00 2001 From: Paolo Dettori Date: Tue, 7 Jul 2026 22:06:53 -0400 Subject: [PATCH 1/3] docs: Add SandboxTransport remote-exec design spec Design for a pluggable harness-sandbox exec seam: a SandboxTransport interface with the existing kubectl-exec kept as the local fast path and a new Redis Streams per-sandbox request/response channel for sandboxes that can only dial outward. Sequences rollout by trust level and defers untrusted-BYO (SPIFFE reverse-tunnel, Approach B) to the same seam. Assisted-By: Claude (Anthropic AI) Signed-off-by: Paolo Dettori --- ...07-sandbox-transport-remote-exec-design.md | 260 ++++++++++++++++++ 1 file changed, 260 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-07-sandbox-transport-remote-exec-design.md diff --git a/docs/superpowers/specs/2026-07-07-sandbox-transport-remote-exec-design.md b/docs/superpowers/specs/2026-07-07-sandbox-transport-remote-exec-design.md new file mode 100644 index 0000000..4d5cec1 --- /dev/null +++ b/docs/superpowers/specs/2026-07-07-sandbox-transport-remote-exec-design.md @@ -0,0 +1,260 @@ +# SandboxTransport: pluggable exec seam + Redis remote sandbox + +**Date:** 2026-07-07 +**Status:** Design — approved for planning +**Scope:** The interface between the harness and the sandbox(es), enabling sandboxes to run outside the harness's cluster (self-hosted / reachable-only-outbound), while keeping the in-cluster path unchanged. + +## 1. Problem & goal + +Today the harness executes every Pi tool call (read / write / bash / grep) inside a +sandbox pod via `kubectl exec` (`packages/k8s-sandbox/src/exec.ts`, +`ExecInPod = (command, opts) => Promise<{ stdout, exitCode }>`). The **harness +initiates the connection into the pod through the kube API**. This only works when +the harness can reach the sandbox's API server, which rules out sandboxes behind +NAT, on-prem, on a laptop, or in another cloud. + +**Goal:** let a sandbox live anywhere by inverting connectivity — the sandbox dials +*out* to a broker the harness also talks to — without changing the Pi orchestration +loop, the session backend, or the leaf queue. + +### Drivers (priority order) + +1. **Bring-your-own (untrusted 3rd-party) sandbox** — external parties host their own + sandbox and register it. *Top priority.* +2. **Decoupling / heterogeneity** — a clean harness↔sandbox contract so sandboxes can + be non-k8s runtimes (VM, Firecracker, remote Docker). +3. **Bursting to external compute** — latency-tolerant overflow onto owned capacity. +4. **Reachability** — sandboxes the harness cannot dial into. + +### Key decisions (settled during brainstorming) + +- **Brain stays central.** The Pi loop + LLM calls remain in the harness; only command + execution is delegated. This is the *trust-correct* choice for driver #1: the LLM key + and control loop never leave the harness; an untrusted sandbox only ever receives + commands and returns bytes. +- **Mirror threat acknowledged.** A malicious sandbox can return poisoned/oversized + stdout into the model's context, and anything the brain sends (a token written to a + file, injected env) is exposed to the sandbox. The return path and what-crosses are + security surfaces, addressed below (output cap; trust-sequenced rollout). +- **Latency posture: mixed.** Transport must degrade gracefully — keep `kubectl-exec` + as the fast in-cluster implementation, add a remote implementation behind one + interface. +- **The queue/worker-with-Redis pattern already exists in-repo** (`@sh/work-queue`, + Redis Streams consumer group with `claim`/`ack`/`touch`/`xAutoClaim`/dead-letter) but + at the **leaf** level (whole independent runs, load-balanced). The exec seam is + **stateful and ordered** (persistent shell cwd/env, filesystem), so it needs a + **per-sandbox, single-worker request/response channel**, not the shared consumer + group. Same Redis primitives, different topology. + +## 2. Architecture + +``` +harness (central brain, Pi loop + LLM) + │ + │ SandboxTransport.exec(cmd, {stdin,onData,signal,timeout}) ← the ONE seam + │ + ├─ [local] KubectlTransport (existing kubectlExecInPod, unchanged) + └─ [remote] RedisChannelTransport (new, harness side of the RPC) + │ + │ sbx::req (harness → worker) Redis Streams + │ sbx::res (worker → harness) + ▼ + @sh/sandbox-worker (new; runs INSIDE the sandbox, dials OUT to Redis) + │ + └─ executes bash -c locally, streams frames back +``` + +### Component boundaries + +| Unit | Purpose | Depends on | +|------|---------|-----------| +| `SandboxTransport` (interface) | The exec seam Pi sees. No transport knowledge above it. | — | +| `KubectlTransport` | Local/in-cluster fast path. Rename of today's `kubectlExecInPod`. | kubectl | +| `RedisChannelTransport` | Harness side: turn one `exec()` into req/res frames + correlation. | `@sh/work-queue` primitives, redis | +| `@sh/sandbox-worker` | Sandbox side: claim req frames, run locally, stream res frames, honor abort. | redis, local shell | +| `select-sandbox` (existing) | Now returns a **transport**, not a pod name. Chooses local vs redis per lease. | pool/lease logic | + +### The interface (fixed core) + +```ts +interface SandboxTransport { + exec(command: string, opts?: { + stdin?: Buffer; + onData?: (chunk: Buffer) => void; + signal?: AbortSignal; + timeout?: number; // seconds + }): Promise<{ stdout: Buffer; exitCode: number | null }>; + close(): Promise; +} +``` + +This is today's `ExecInPod` capability surface, verbatim, plus `close()`. Pi depends on +all of it (streaming via `onData`; abort via `signal`; `timeout`; `stdin` for writes), +so the interface must preserve all of it. + +### Architectural invariants + +- **Single worker per sandbox.** The `sbx::req|res` streams are per-sandbox and + owned by exactly one worker. This preserves shell/cwd/env/filesystem ordering, which + the load-balanced leaf queue explicitly does not guarantee. `xAutoClaim` reclaim is + used **only for crash recovery of that one worker**, never to load-balance across + workers. +- **One exec in flight per sandbox.** Mirrors `persistent-exec.ts` today; bounds memory + and makes req order == execution order. +- **Everything above `select-sandbox` is transport-blind.** `run-leaf`, `run-turn`, + `converge` receive a `SandboxTransport` and never learn how bytes reach the sandbox. + `converge` (repo sync) already goes through exec, so it rides the transport for free. + +### What does NOT change + +The Pi orchestration loop, `run-turn`, the session backend (`RedisSessionBackend`), and +the leaf queue (`@sh/work-queue`). The change slots strictly *below* the current +`ExecInPod` call sites (`converge.ts`, `run-leaf.ts`, `run-turn.ts`, `select-sandbox.ts`). + +## 3. Wire protocol (`sbx::req` / `sbx::res`) + +Reuses `packages/k8s-sandbox/src/framing.ts` (`wrapCommand` / `FrameParser`, +`\x01`-marker + base64 body, exit via `${PIPESTATUS[0]}`) inside the worker: the worker +runs the existing persistent bash locally and Redis simply replaces the kubectl +stdin/stdout pipe. The one extension: the persistent channel is one-shot per command +(no `onData` streaming) — for a *remote* sandbox we want live output, so the protocol +adds incremental `chunk` frames. + +**Correlation & ordering.** Each `exec()` gets a monotonic `reqId` (like +`persistent-exec`'s `seq`). One exec in flight ⇒ req order == execution order. `reqId` +doubles as the idempotency key. + +**Frames** (JSON envelopes; byte payloads base64-encoded): + +| Direction | `kind` | Fields | Purpose | +|-----------|--------|--------|---------| +| harness → `req` | `exec` | `reqId, command, stdinB64?, timeout, streaming` | run one command | +| harness → `req` | `abort` | `reqId` | cancel the in-flight exec | +| worker → `res` | `chunk` | `reqId, dataB64` | streamed stdout (0..n; only when `streaming`) | +| worker → `res` | `end` | `reqId, exitCode` | terminal success | +| worker → `res` | `error` | `reqId, message` | channel/exec failure → harness fallback or reject | + +- **Non-streaming ops** (read/write) emit a single `end` carrying full stdout — the + current one-shot frame. +- **Streaming ops** (bash/grep) emit `chunk`* then `end`; the harness replays each + `chunk` into `opts.onData`, matching today's `ExecInPod` contract byte-for-byte. + +**Abort & timeout.** The worker's req-read loop runs independently of the in-flight +child, so an `abort` frame is seen mid-exec → local `SIGKILL` (same effect as today's +`signal` → `child.kill`). Timeout is enforced **on both ends**: the worker kills locally +at `timeout`; the harness has its own deadline and synthesizes a timeout `error` if no +`end` arrives — it never hangs on a silent or malicious worker. + +**At-least-once → dedup.** The `req` stream uses a consumer group with **one consumer +(the worker)**; `xAutoClaim` reclaims an unacked req only for worker crash recovery. +Because a redelivered write is not idempotent, the worker keeps a bounded last-N +`reqId → end` cache and **re-emits the cached result instead of re-running**. +*Honest limitation:* if the worker died mid-write, exactly-once is impossible — the +contract is at-least-once + dedup-by-`reqId`, and partial filesystem effects on crash +are possible (same risk class as a leaf re-run today). + +**Poisoned-output defense (untrusted return path).** The harness enforces a +**total-output cap per exec**; on exceed it sends `abort`, truncates, and surfaces an +`[output truncated]` marker to Pi. This is the concrete mitigation for the driver-#1 +threat of a malicious sandbox flooding the model's context. + +## 4. The sandbox worker & transport selection + +### `@sh/sandbox-worker` (new; runs inside the sandbox, dials out) + +A thin, single-purpose process — a pump between Redis and one local persistent bash. +It holds **no LLM key and no orchestration**; it only executes commands and returns +bytes (the property that made "central brain" trust-correct). + +1. On start: read `SANDBOX_ID` + `REDIS_URL` from env; `ensureGroup()` on + `sbx::req`; connect; write + heartbeat a registration record. +2. Loop: `claim` next req frame (blocking read + `xAutoClaim` reclaim for + self-recovery). +3. `exec` frame → feed `wrapCommand(reqId, command, stdin)` into the local persistent + bash (reuse `framing.ts`); pipe the framed stdout back out as `chunk`/`end` frames on + `sbx::res`; `ack` after `end`. +4. `abort` frame → `SIGKILL` the local child. +5. Maintain the bounded `reqId → end` dedup cache; re-emit on redelivery. + +### Transport selection (`select-sandbox`, modified) + +`select-sandbox` already leases a sandbox from the pool via Redis. It now returns a +`SandboxTransport` instead of a pod name/config: + +``` +lease sandbox → inspect its registration record → + kind: "kubectl" → KubectlTransport(config) (local, fast — unchanged path) + kind: "redis" → RedisChannelTransport(sandboxId) (remote worker) +``` + +The pool's lease record gains a `transport: "kubectl" | "redis"` field (and, for redis, +the `sandboxId`). + +### Registration (v1: trusted-unreachable) + +A remote worker, on startup, writes a registration record to Redis +(`sandbox:registry:` with `transport:"redis"`, capacity, labels) and heartbeats it. +The pool's lease logic considers registered remote sandboxes alongside in-cluster pods; +missed heartbeats de-register. **This registry is the seam where Approach B / SPIFFE +authenticated registration later slots in for untrusted BYO — same record, stronger +identity.** + +## 5. Error handling & lifecycle + +| Failure | Detection | Behavior | +|---------|-----------|----------| +| Remote worker crash mid-exec | req frame unacked; `xAutoClaim` after `minIdleMs` | Redelivered to restarted worker. Dedup cache re-emits if already completed; else re-run (at-least-once). Partial-write risk documented. | +| Worker never starts / gone | missed heartbeat; harness deadline with no `end` | Lease marked unhealthy, sandbox evicted from pool; in-flight `exec` rejects with timeout `error`. Leaf retry re-leases a healthy sandbox. | +| Redis unreachable (harness) | connect/command error | `RedisChannelTransport.exec` rejects; leaf fails and re-queues via existing leaf `WorkQueue`. Redis is a documented availability dependency for remote execs. | +| Redis unreachable (worker) | connect error | Worker exits non-zero; supervisor restarts it; registry heartbeat lapses meanwhile. | +| In-flight exec exceeds timeout | dual deadline (§3) | Worker SIGKILLs local child; harness synthesizes timeout — never hangs on a silent worker. | +| Output cap exceeded (poisoned/runaway) | harness byte counter on `res` | Harness sends `abort`, truncates, surfaces `[output truncated]` to Pi. | +| Abort races with `end` | `reqId` correlation | Late `end` for an aborted `reqId` is dropped; abort for an already-ended `reqId` is a no-op. | +| Stale frames from a prior lease | `reqId`/session scoping | Worker resets stream position on (re)registration; harness ignores frames whose `reqId` predates the current transport instance. | + +**Lifecycle.** The transport is created at lease time; `close()` on leaf completion +(dispose local bash / stop reading `res`). The worker's lifetime is independent — it +outlives individual leases and serves whatever the pool assigns, matching the existing +shared-pool model. + +**Backpressure.** One exec in flight per sandbox (natural from a single persistent bash) +bounds memory; `chunk` size is capped so a single `res` entry stays small. + +## 6. Testing + +| Layer | What | How | +|-------|------|-----| +| Pure unit | Frame protocol, dedup cache, output-cap counter, timeout math, transport-selection logic | Plain vitest, no I/O. Extends existing `framing.ts` tests. | +| Contract | `RedisChannelTransport` + `@sh/sandbox-worker` against a real Redis, worker pointed at a local bash | Round-trip every op (read/write/bash/grep), abort mid-stream, timeout, redelivery→dedup, output-cap. Core suite. | +| Conformance | The *same* battery run against **both** `KubectlTransport` and `RedisChannelTransport` | One shared spec proving identical `SandboxTransport` contract — this is what makes them safely swappable (driver #2). | +| Live gate | Real worker pod dialing out to in-cluster Redis; one leaf end-to-end on Kind, then OCP | Follows the existing leaf-smoke pattern; manifest-shape vitest parses the worker Deployment YAML directly (no kustomize in CI). | + +## 7. Rollout & scope + +**Flagged, off by default, sequenced by trust:** + +1. `SandboxTransport` interface + `KubectlTransport` rename. **Pure refactor, zero + behavior change** — ships first, provably inert (conformance suite green, existing + e2e green). +2. `@sh/sandbox-worker` + `RedisChannelTransport` + registry, behind + `SH_REMOTE_SANDBOX=off` by default. No effect until a remote worker registers. +3. Enable on Kind/OCP for **trusted-unreachable** workers (Redis ACL per `sbx::*` + prefix + heartbeat registration). +4. **Deferred (explicit non-goal for this spec):** untrusted BYO isolation → Approach B + (SPIFFE-authenticated registration + per-connection identity via SPIRE/AuthBridge), + slotting into the same registry seam. + +**Scope boundaries (YAGNI):** no multi-worker-per-sandbox, no cross-region Redis, no +reverse-tunnel / gRPC (Approach B), no changes to the Pi loop or leaf queue. This spec +is exactly: the interface + one remote transport + the worker + the registry. + +## 8. Deferred: Approach B (for untrusted BYO) + +Recorded so the sequencing is explicit, not lost. When untrusted 3rd-party sandboxes +(driver #1) become real, Redis-as-shared-multi-tenant-bus is the weak point (coarse +ACLs; a scoping bug = cross-tenant exec injection). Approach B — the sandbox holds a +persistent outbound gRPC/WebSocket stream to a harness-side relay, with **per-connection +identity via mTLS/SPIFFE** (infra already operated via SPIRE + AuthBridge) — answers the +trust concern directly and gives protocol-native streaming/backpressure/abort. It plugs +into the **same registration seam** defined in §4, behind the **same `SandboxTransport` +interface** from §2, so adopting it is additive, not a rewrite. From 763cc93698380c9a96c13eabd6221173365867c8 Mon Sep 17 00:00:00 2001 From: Paolo Dettori Date: Wed, 8 Jul 2026 16:03:42 -0400 Subject: [PATCH 2/3] docs: revise SandboxTransport spec to language-neutral gRPC Replace the Redis-Streams remote-exec transport with a language-neutral protobuf contract served over one gRPC bidi stream on :443, and relocate the spec to the canonical docs/specs/ directory. - Protobuf IDL as the contract; any-language workers (Go reference worker) - gRPC-native HTTP/2 over :443 (firewall-friendly), single bidi Attach stream - Single-replica, presence-only relay; matching stays in the existing pool - Per-sandbox bearer token at the edge; SPIFFE/mTLS deferred - Frame semantics (req_id / dedup / dual-timeout / output-cap) carried verbatim - Drop RedisChannelTransport and @sh/sandbox-worker; keep the SandboxTransport interface + KubectlTransport rename refactor Assisted-By: Claude (Anthropic AI) Signed-off-by: Paolo Dettori --- ...026-07-08-sandbox-transport-grpc-design.md | 440 ++++++++++++++++++ ...07-sandbox-transport-remote-exec-design.md | 260 ----------- 2 files changed, 440 insertions(+), 260 deletions(-) create mode 100644 docs/specs/2026-07-08-sandbox-transport-grpc-design.md delete mode 100644 docs/superpowers/specs/2026-07-07-sandbox-transport-remote-exec-design.md diff --git a/docs/specs/2026-07-08-sandbox-transport-grpc-design.md b/docs/specs/2026-07-08-sandbox-transport-grpc-design.md new file mode 100644 index 0000000..1374435 --- /dev/null +++ b/docs/specs/2026-07-08-sandbox-transport-grpc-design.md @@ -0,0 +1,440 @@ +# SandboxTransport: language-neutral remote exec over gRPC + +**Date:** 2026-07-08 +**Status:** Design — approved for planning +**Scope:** The interface between the harness and the sandbox(es), enabling sandboxes to +run outside the harness's cluster (self-hosted / laptop / on-prem / other cloud / +bring-your-own), while keeping the in-cluster `kubectl-exec` path unchanged. +**Supersedes:** the Redis-Streams transport previously proposed in this PR +(`2026-07-07-sandbox-transport-remote-exec-design.md`). See §12 for why. + +## 1. Problem & goal + +Today the harness executes every Pi tool call (read / write / bash / grep) inside a +sandbox pod via `kubectl exec` (`packages/k8s-sandbox/src/exec.ts`, +`ExecInPod = (command, opts) => Promise<{ stdout, exitCode }>`). The **harness initiates +the connection into the pod through the kube API**. This only works when the harness can +reach the sandbox's API server, which rules out sandboxes behind NAT, on-prem, on a +laptop, or in another cloud. + +**Goal:** let a sandbox live anywhere by inverting connectivity — the sandbox dials *out* +to a broker the harness also talks to — with a protocol that is **language-neutral** +(any runtime can host a worker) and **firewall-friendly** (a single outbound TLS +connection on `:443`), without changing the Pi orchestration loop, the session backend, +or the leaf queue. + +### Drivers (priority order) + +1. **Bring-your-own (untrusted 3rd-party) sandbox** — external parties host their own + sandbox and register it. *Top priority.* +2. **Decoupling / heterogeneity** — a clean, language-independent harness↔sandbox + contract so a sandbox can be any runtime (VM, Firecracker, remote Docker) in any + language. +3. **Bursting to external compute** — latency-tolerant overflow onto owned capacity. +4. **Reachability** — sandboxes the harness cannot dial into. + +## 2. Key decisions (settled during brainstorming) + +- **Brain stays central.** The Pi loop + LLM calls remain in the harness; only command + execution is delegated. This is the *trust-correct* choice for driver #1: the LLM key + and control loop never leave the harness; an untrusted sandbox only ever receives + commands and returns bytes. +- **The contract is a Protobuf IDL, not a TypeScript interface.** Any language with gRPC + codegen (Go, Python, Rust, Java, TS, …) can implement a worker. The `.proto` is the + source of truth; TypeScript is just one generated client. This is what makes driver #2 + real. +- **gRPC-native over HTTP/2 on `:443`.** One outbound TLS connection carrying a + full-duplex bidirectional stream. HTTP/2-on-443 traverses most modern egress and NAT. + *(Connect / HTTP-1.1 fallback was considered and rejected — see §5 — because our + streaming core is full-duplex, which requires HTTP/2 regardless.)* +- **A single-replica relay, presence-only.** A new in-cluster process bridges the + worker's outbound stream to the harness's in-cluster calls. It does **not** own + matching or leasing — it mirrors connected workers into the **existing sandbox pool**, + and the existing `select-sandbox` logic does the matching unchanged. +- **One reference worker, in Go.** A single static binary with no runtime dependencies, + droppable into any sandbox image — and the honest proof that the contract is genuinely + language-neutral rather than secretly TS-shaped. +- **Per-sandbox bearer token at the edge.** The worker authenticates on connect; + SPIFFE/mTLS for untrusted BYO upgrades into the *same* seam later. +- **Latency posture: mixed.** `kubectl-exec` stays as the fast in-cluster implementation; + the remote path is added behind the same interface and degrades gracefully. + +## 3. Architecture + +``` +Sandbox (laptop / on-prem / other cloud / same cluster) + worker (Go static binary) — implements SandboxWorker.Attach client + │ outbound only, TLS :443, HTTP/2 + │ ServerFrame{Exec|Abort} ↓ WorkerFrame{Hello|Heartbeat|Chunk|End|Error} ↑ + ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ Kubernetes │ +│ relay (Deployment, 1 replica) │ +│ SandboxWorker.Attach ← workers dial in (bidi stream) │ +│ SandboxExec.Exec/Abort ← harness calls (in-cluster) │ +│ in-memory session table: sandbox_id → live Attach stream │ +│ mirrors presence → existing Redis sandbox pool │ +│ │ in-cluster gRPC │ presence write/remove │ +│ ▼ ▼ │ +│ Harness (central brain, Pi loop + LLM) Redis sandbox pool │ +│ SandboxTransport.exec(...) ← the ONE seam │ +│ ├─ [local] KubectlTransport (existing kubectlExecInPod) │ +│ └─ [remote] GrpcRelayTransport (new; calls SandboxExec) │ +│ select-sandbox → leases pod OR remote record → returns a transport │ +└──────────────────────────────────────────────────────────────────────┘ +``` + +### Component boundaries + +| Unit | Purpose | Depends on | +|------|---------|-----------| +| `SandboxTransport` (interface) | The exec seam Pi sees. No transport knowledge above it. | — | +| `KubectlTransport` | Local/in-cluster fast path. Rename of today's `kubectlExecInPod`. | kubectl | +| `GrpcRelayTransport` | Harness side: turn one `exec()` into a `SandboxExec.Exec` call + stream reassembly + correlation. | relay (in-cluster gRPC) | +| `relay` | Bridge worker's outbound `Attach` stream ↔ harness `Exec`; mirror presence into the pool. | gRPC, Redis pool | +| Go worker (reference) | Sandbox side: dial `Attach`, run commands locally, stream frames, honor abort. | local shell | +| `select-sandbox` (existing) | Now returns a **transport**, not a pod name. Sees pods + remote records; leases least-loaded. | pool/lease logic | + +### The harness-facing interface (unchanged from PR #78) + +```ts +interface SandboxTransport { + exec(command: string, opts?: { + stdin?: Buffer; + onData?: (chunk: Buffer) => void; + signal?: AbortSignal; + timeout?: number; // seconds + }): Promise<{ stdout: Buffer; exitCode: number | null }>; + close(): Promise; +} +``` + +This is today's `ExecInPod` capability surface verbatim, plus `close()`. Pi depends on all +of it (streaming via `onData`; abort via `signal`; `timeout`; `stdin` for writes), so the +interface preserves all of it. `GrpcRelayTransport` and `KubectlTransport` are two +implementations of this one interface. + +### Architectural invariants + +- **Single worker per sandbox; one exec in flight per sandbox.** Preserves + shell/cwd/env/filesystem ordering and makes req order == execution order. Mirrors + today's `persistent-exec.ts`. +- **Everything above `select-sandbox` is transport-blind.** `run-leaf`, `run-turn`, + `converge` receive a `SandboxTransport` and never learn how bytes reach the sandbox. + `converge` (repo sync) already goes through exec, so it rides the transport for free. +- **The relay is a stateless byte-bridge.** It parses only enough to route by + `sandbox_id`; the exec is effectively end-to-end between harness and worker. + +### What does NOT change + +The Pi orchestration loop, `run-turn`, the session backend (`RedisSessionBackend`), the +leaf queue (`@sh/work-queue`), and the sandbox **pool/lease** logic. The change slots +strictly *below* the current `ExecInPod` call sites (`converge.ts`, `run-leaf.ts`, +`run-turn.ts`, `select-sandbox.ts`). + +## 4. The Protobuf contract (`sandbox/v1`) + +Two services. No registry service and no `Selector` message — matching stays in the +harness pool (§6). + +```protobuf +syntax = "proto3"; +package sandbox.v1; + +// Worker (any language) dials the relay and holds ONE bidi stream. +// worker→relay carries results/liveness; relay→worker carries commands. +service SandboxWorker { + rpc Attach(stream WorkerFrame) returns (stream ServerFrame); +} + +// Harness asks the relay to run a command on a connected sandbox and +// streams the output back. The relay bridges to the worker's Attach +// stream, keyed by sandbox_id. In-cluster only. +service SandboxExec { + rpc Exec(ExecRequest) returns (stream ExecEvent); // server-streaming output + rpc Abort(AbortRequest) returns (AbortResponse); +} + +// ---- worker → relay ---- +message WorkerFrame { + oneof msg { + Hello hello = 1; // sent first: identity + capabilities + Heartbeat heartbeat = 2; // liveness; also keeps NAT/proxy state open + Chunk chunk = 3; // streamed stdout for an exec + End end = 4; // terminal success for an exec + ExecError error = 5; // exec/channel failure + } +} + +// ---- relay → worker ---- +message ServerFrame { + oneof msg { + Exec exec = 1; // run one command + Abort abort = 2; // cancel an in-flight exec + } +} + +message Hello { + string sandbox_id = 1; + map labels = 2; // team=alpha, env=prod, region=us-east + repeated string capabilities = 3; // ["python3","kubectl","gpu","node20"] + string image = 4; // sandbox image tag/digest + string arch = 5; // amd64 | arm64 + uint32 capacity_max = 6; // max concurrent execs the worker allows + string trust = 7; // "trusted" | "untrusted" +} + +message Heartbeat {} // liveness/keepalive only; harness owns lease counts + +message Exec { + uint64 req_id = 1; // monotonic; correlation + dedup key + string command = 2; + bytes stdin = 3; // raw bytes (protobuf is binary — no base64) + uint32 timeout_s = 4; + bool streaming = 5; // true for bash/grep; false for read/write +} +message Abort { uint64 req_id = 1; } +message Chunk { uint64 req_id = 1; bytes data = 2; } // 0..n, only if streaming +message End { uint64 req_id = 1; sint32 exit_code = 2; } // exit_code < 0 = signal/none +message ExecError { uint64 req_id = 1; string message = 2; } + +// ---- harness ↔ relay (SandboxExec) ---- +message ExecRequest { string sandbox_id = 1; Exec exec = 2; } // sandbox_id from the lease +message ExecEvent { oneof event { Chunk chunk = 1; End end = 2; ExecError error = 3; } } +message AbortRequest { string sandbox_id = 1; uint64 req_id = 2; } +message AbortResponse {} +``` + +**Notes** + +- **Binary, not base64.** `bytes stdin` / `bytes data` ride protobuf's native binary + encoding — smaller and faster than the JSON+base64 frames of the superseded design. +- **The worker does not self-report `capacity_in_use`.** The harness owns the leases (the + pool ZSET already tracks how many it has granted each sandbox), so `Hello` carries only + `capacity_max`, and `Heartbeat` is pure liveness/keepalive. + +## 5. Connectivity: one bidi `Attach` stream, worker-dialed + +The constraint: the **worker can only dial out**, but the **harness must push commands to +the worker**. A single **bidirectional streaming RPC where the worker is the client** +resolves this — commands flow to the worker on the *server→client* half of the stream the +worker itself opened: + +``` +Worker (client, dials OUT :443) Relay (server, in-cluster) + │ Attach(stream WorkerFrame) ───────────────────────▶│ worker→relay: results & liveness + │◀─────────────────────────── returns (stream ServerFrame) relay→worker: commands +``` + +The worker opens one long-lived bidi stream and **never listens for inbound +connections**. Commands arrive on the response side of its own outbound call — the +outbound-only property we want, expressed in a standard RPC primitive. + +**Why gRPC-native and not Connect.** Connect's headline advantage is a plain-HTTP/1.1 +fallback that traverses HTTP-inspecting proxies. But **full-duplex bidi streaming +requires HTTP/2 regardless of framework** — Connect only offers unary + server-streaming +over HTTP/1.1. Since our core *is* a full-duplex stream, Connect buys us little for the +part that matters while adding a second toolchain. We therefore use gRPC-native over +HTTP/2 on `:443`. If a concrete "must traverse HTTP/1.1-only proxy" requirement ever +appears, the fallback is to decompose `Attach` into a server-streaming "receive commands" +call plus unary "post results" calls — a future protocol variant behind the same seam, +explicitly out of scope here. + +## 6. The relay (single-replica, presence-only) + +A new in-cluster `Deployment`, **one replica**. It is a matchmaker-free byte bridge: + +1. **Accepts `Attach`.** On `Hello`, validates the bearer token ↔ `sandbox_id` binding + (§8), then parks the live stream in an in-memory session table `sandbox_id → stream`. +2. **Mirrors presence into the existing Redis sandbox pool.** It writes a lightweight + record (`sandbox_id`, labels, capabilities, `capacity_max`, `transport:"grpc"`) into + the same pool store `select-sandbox` already reads, and **removes it when the stream + closes**. The live `Attach` stream *is* the registration — no separate heartbeat key, + no reaper. +3. **Routes `SandboxExec.Exec`.** Looks up the live stream by `sandbox_id`, sends + `ServerFrame{Exec}`, and forwards the worker's `Chunk`/`End`/`Error` back as the + `stream ExecEvent` response. `Abort` is routed the same way. + +**Why single-replica.** With exactly one replica there is no "which replica holds this +worker's stream" problem, so the relay needs **no presence glue** (no K8s-API watch, no +Redis pub/sub) beyond the pool mirror above. This is the deliberate day-one simplification +(§13). + +**Relay restart = in-flight execs fail.** When the single replica restarts, all parked +streams drop; workers reconnect and re-register; any `Exec` in flight fails. This degrades +into existing recovery: the leaf fails and re-leases a healthy sandbox on retry (see the +error table, §10). There is **no mid-exec durability** — stated plainly so nobody expects +it. + +### Harness integration: `select-sandbox` barely moves + +`select-sandbox` already leases from the pool via Redis. It now: + +``` +lease from pool (in-cluster pods + mirrored remote records) → + record.transport == "kubectl" → KubectlTransport(config) (local, fast — unchanged) + record.transport == "grpc" → GrpcRelayTransport(sandbox_id) (routes via the relay) +``` + +The pool remains the single source of truth for matching; a remote sandbox is just +another pool entry that happens to resolve to a different transport. + +## 7. The Go worker (reference implementation) + +A thin, single-purpose static binary. It holds **no LLM key and no orchestration** — it +only executes commands and returns bytes (the property that makes "central brain" +trust-correct). It: + +1. On start: read `SANDBOX_ID`, `RELAY_ADDR`, `SANDBOX_TOKEN` from env; dial + `SandboxWorker.Attach` over TLS `:443` with `Authorization: Bearer `. +2. Send `Hello` (id, labels, capabilities, `capacity_max`, `trust`). +3. Loop on `ServerFrame`: + - `Exec` → run `bash -c ` locally (feeding `stdin`), stream stdout back as + `Chunk`* then `End{exit_code}` when `streaming`; otherwise a single `End` carrying + full stdout. Enforce `timeout_s` locally (SIGKILL on expiry). + - `Abort` → SIGKILL the in-flight child for that `req_id`. +4. Send `Heartbeat` periodically (liveness + NAT/proxy keepalive). +5. Maintain a bounded `req_id → End` dedup cache; **re-emit the cached result** instead of + re-running on a redelivered `req_id` (§9). + +Persistent-shell semantics (a single long-lived `bash` preserving cwd/env across execs) +are an implementation choice inside the worker; the wire contract only requires ordered, +one-in-flight execs. + +## 8. Wire semantics (correlation, dedup, timeout, output cap, abort) + +The frame *semantics* are carried from the superseded design verbatim — only the encoding +(protobuf) and transport (gRPC bidi) changed. + +- **Correlation & ordering.** Each `exec()` gets a monotonic `req_id`. One exec in flight + ⇒ req order == execution order. `req_id` doubles as the idempotency key. +- **Streaming vs one-shot.** Streaming ops (bash/grep) emit `Chunk`* then `End`; the + harness replays each `Chunk` into `opts.onData`, matching today's `ExecInPod` contract + byte-for-byte. Non-streaming ops (read/write) emit a single `End` with full stdout. +- **At-least-once → dedup.** A reconnect mid-exec can redeliver a command. The worker's + bounded `req_id → End` cache re-emits the cached terminal result rather than re-running. + *Honest limitation:* if the worker died mid-write, exactly-once is impossible — the + contract is at-least-once + dedup-by-`req_id`, and partial filesystem effects on crash + are possible (same risk class as a leaf re-run today). +- **Dual-ended timeout.** The worker kills its local child at `timeout_s`; the harness has + its own deadline and synthesizes a timeout `error` if no `End` arrives — it never hangs + on a silent or malicious worker. +- **Poisoned-output defense.** The harness enforces a **total-output cap per exec**; on + exceed it issues `Abort`, truncates, and surfaces an `[output truncated]` marker to Pi. + This is the concrete mitigation for the driver-#1 threat of a malicious sandbox flooding + the model's context. `Chunk` size is capped so a single frame stays small (backpressure). +- **Abort/end races.** A late `End` for an aborted `req_id` is dropped; an `Abort` for an + already-ended `req_id` is a no-op. + +## 9. Security & reachability + +**Mode A — public relay on `:443` (the only mode day-one).** The worker dials +`https://relay.` with TLS and `Authorization: Bearer `. The +relay validates the token ↔ `sandbox_id` binding on `Attach` before parking the stream. +The worker-facing `Attach` endpoint is the **single public attack surface**; the +harness-facing `SandboxExec` service is **in-cluster only**, guarded by NetworkPolicy. + +| Property | Value | +|----------|-------| +| Inbound rules on sandbox | none (outbound only) | +| Egress required | `:443` only | +| Encryption | TLS 1.3 | +| Worker identity (day-one) | per-sandbox bearer token, scoped to `sandbox_id` | +| Public attack surface | relay `Attach` on `:443` | + +**Upgrade path for untrusted BYO.** SPIFFE/SPIRE **mTLS** with per-connection identity +slots into the *same* `Attach` endpoint — same protocol, stronger credential on the TLS +handshake, no wire change. Deferred (§13); the bearer token is the seam it replaces. + +**Reachability is pluggable beneath the RPC.** A private-mesh mode (self-hosted +**Headscale** / plain **WireGuard**) for fleets that refuse any public relay endpoint can +be added later without a protocol change. No paid external dependency is introduced; +Tailscale is explicitly out of scope. + +## 10. Error handling & lifecycle + +| Failure | Detection | Behavior | +|---------|-----------|----------| +| Worker disconnects (crash / network) | `Attach` stream closes | Presence record removed from pool; worker reconnects and re-registers. In-flight `Exec` fails; leaf retry re-leases a healthy sandbox. | +| Command redelivered after reconnect | duplicate `req_id` at worker | Dedup cache re-emits cached `End`; else re-run (at-least-once). Partial-write risk documented (§8). | +| Relay restart | all parked streams drop | Workers reconnect; in-flight execs fail → leaf retry. No mid-exec durability. | +| Worker never connects / gone | no live stream for `sandbox_id`; harness deadline | Pool entry absent or evicted; `Exec` rejects; leaf retry re-leases elsewhere. | +| In-flight exec exceeds timeout | dual deadline (§8) | Worker SIGKILLs local child; harness synthesizes timeout — never hangs on a silent worker. | +| Output cap exceeded (poisoned/runaway) | harness byte counter on `ExecEvent` | Harness `Abort`s, truncates, surfaces `[output truncated]` to Pi. | +| Abort races with `End` | `req_id` correlation | Late `End` for an aborted `req_id` dropped; `Abort` for an ended `req_id` is a no-op. | +| Bad / missing token | relay validates on `Attach` | Stream rejected before it is parked; no pool entry created. | + +**Lifecycle.** The `GrpcRelayTransport` is created at lease time; `close()` on leaf +completion stops reading the `ExecEvent` stream. The worker's lifetime is independent — it +outlives individual leases and serves whatever the pool assigns, matching the existing +shared-pool model. + +## 11. Testing + +| Layer | What | How | +|-------|------|-----| +| Pure unit | Frame reassembly, dedup cache, output-cap counter, timeout math, transport selection | Plain vitest / Go test, no I/O. | +| Contract | `GrpcRelayTransport` + Go worker against a real relay, worker pointed at a local bash | Round-trip every op (read/write/bash/grep), abort mid-stream, timeout, reconnect→dedup, output-cap. Core suite. | +| Conformance | The *same* battery run against **both** `KubectlTransport` and `GrpcRelayTransport` | One shared spec proving identical `SandboxTransport` contract — this is what makes them safely swappable (driver #2). | +| Live gate | Real Go worker pod dialing the in-cluster relay over `:443`; one leaf end-to-end on Kind, then OCP | Follows the existing leaf-smoke pattern; manifest-shape vitest parses the relay + worker Deployment YAML directly (no kustomize in CI). | + +## 12. Why this supersedes the Redis-Streams design + +The previous revision of this PR inverted connectivity correctly but tied down two things +that limited reach and portability: + +1. **Language lock-in.** The contract was a TypeScript `SandboxTransport` interface with + JSON+base64 frames; a worker had to be TS (or hand-reimplement the frames). Driver #2 + (heterogeneity / BYO in any language) could not be met. +2. **Redis as the wire.** Redis Streams carried the RPC. Redis on its port is commonly + blocked by egress firewalls that allow only `:443`, and it forced a Redis availability + dependency into the exec path. + +This design keeps the outbound-dial insight and the frame *semantics* (`req_id`, dedup, +dual timeout, output cap — §8) **verbatim**, and changes only the encoding and transport: +protobuf over one gRPC bidi stream on `:443`. What survives unchanged from the earlier +work is the `SandboxTransport` interface and the `KubectlTransport` rename (a pure, +behavior-preserving refactor). What is **dropped** is the Redis-specific machinery — the +`RedisChannelTransport` and the TS `@sh/sandbox-worker` are **not built**; Redis returns +to being only the pool/session/leaf-queue substrate, never the exec wire. + +## 13. Rollout & scope + +**Flagged, off by default, sequenced by trust:** + +1. `SandboxTransport` interface + `KubectlTransport` rename. **Pure refactor, zero + behavior change** — ships first, provably inert (conformance suite green, existing e2e + green). +2. `sandbox/v1` proto + relay (single replica) + `GrpcRelayTransport` + Go worker + + presence mirror, behind a flag (`SH_REMOTE_SANDBOX`, off by default). No effect until a + worker connects. +3. Enable on Kind/OCP for **trusted-unreachable** workers (per-sandbox bearer token + + `:443` relay endpoint). +4. **Deferred (explicit non-goals for this spec):** + - Untrusted-BYO isolation → SPIFFE/SPIRE **mTLS** per-connection identity on the same + `Attach` endpoint. + - **Multi-replica relay / HA** → adds a presence map (K8s API or Redis pub/sub) to + route across replicas; unnecessary until relay load or availability demands it. + - **Private-mesh reachability** (Headscale / WireGuard) for fleets that refuse a public + endpoint. + - **Additional-language workers** (e.g. Python for ML/data sandboxes) — the proto + already permits them; only the Go reference worker ships now. + - **HTTP/1.1-only proxy traversal** → the Connect / split-stream variant of §5. + +**Scope boundaries (YAGNI):** no multi-replica relay, no presence glue beyond the pool +mirror, no Tailscale, no registry/matching service, no `Selector`, no changes to the Pi +loop or leaf queue. This spec is exactly: the interface + the `GrpcRelayTransport` + the +single-replica relay + the Go worker + the presence mirror. + +## 14. References + +- Superseded PR revision — `2026-07-07-sandbox-transport-remote-exec-design.md` (Redis + Streams; removed in this update). +- Shared sandbox pool — `2026-07-02-p2-shared-sandbox-pool-design.md` +- Identity spine (SPIFFE/SPIRE + AuthBridge) — `2026-06-26-identity-spine-design.md` +- Protobuf / gRPC — https://protobuf.dev , https://grpc.io +- SPIFFE / SPIRE — https://spiffe.io + +--- + +*Assisted-By: Claude (Anthropic AI) * diff --git a/docs/superpowers/specs/2026-07-07-sandbox-transport-remote-exec-design.md b/docs/superpowers/specs/2026-07-07-sandbox-transport-remote-exec-design.md deleted file mode 100644 index 4d5cec1..0000000 --- a/docs/superpowers/specs/2026-07-07-sandbox-transport-remote-exec-design.md +++ /dev/null @@ -1,260 +0,0 @@ -# SandboxTransport: pluggable exec seam + Redis remote sandbox - -**Date:** 2026-07-07 -**Status:** Design — approved for planning -**Scope:** The interface between the harness and the sandbox(es), enabling sandboxes to run outside the harness's cluster (self-hosted / reachable-only-outbound), while keeping the in-cluster path unchanged. - -## 1. Problem & goal - -Today the harness executes every Pi tool call (read / write / bash / grep) inside a -sandbox pod via `kubectl exec` (`packages/k8s-sandbox/src/exec.ts`, -`ExecInPod = (command, opts) => Promise<{ stdout, exitCode }>`). The **harness -initiates the connection into the pod through the kube API**. This only works when -the harness can reach the sandbox's API server, which rules out sandboxes behind -NAT, on-prem, on a laptop, or in another cloud. - -**Goal:** let a sandbox live anywhere by inverting connectivity — the sandbox dials -*out* to a broker the harness also talks to — without changing the Pi orchestration -loop, the session backend, or the leaf queue. - -### Drivers (priority order) - -1. **Bring-your-own (untrusted 3rd-party) sandbox** — external parties host their own - sandbox and register it. *Top priority.* -2. **Decoupling / heterogeneity** — a clean harness↔sandbox contract so sandboxes can - be non-k8s runtimes (VM, Firecracker, remote Docker). -3. **Bursting to external compute** — latency-tolerant overflow onto owned capacity. -4. **Reachability** — sandboxes the harness cannot dial into. - -### Key decisions (settled during brainstorming) - -- **Brain stays central.** The Pi loop + LLM calls remain in the harness; only command - execution is delegated. This is the *trust-correct* choice for driver #1: the LLM key - and control loop never leave the harness; an untrusted sandbox only ever receives - commands and returns bytes. -- **Mirror threat acknowledged.** A malicious sandbox can return poisoned/oversized - stdout into the model's context, and anything the brain sends (a token written to a - file, injected env) is exposed to the sandbox. The return path and what-crosses are - security surfaces, addressed below (output cap; trust-sequenced rollout). -- **Latency posture: mixed.** Transport must degrade gracefully — keep `kubectl-exec` - as the fast in-cluster implementation, add a remote implementation behind one - interface. -- **The queue/worker-with-Redis pattern already exists in-repo** (`@sh/work-queue`, - Redis Streams consumer group with `claim`/`ack`/`touch`/`xAutoClaim`/dead-letter) but - at the **leaf** level (whole independent runs, load-balanced). The exec seam is - **stateful and ordered** (persistent shell cwd/env, filesystem), so it needs a - **per-sandbox, single-worker request/response channel**, not the shared consumer - group. Same Redis primitives, different topology. - -## 2. Architecture - -``` -harness (central brain, Pi loop + LLM) - │ - │ SandboxTransport.exec(cmd, {stdin,onData,signal,timeout}) ← the ONE seam - │ - ├─ [local] KubectlTransport (existing kubectlExecInPod, unchanged) - └─ [remote] RedisChannelTransport (new, harness side of the RPC) - │ - │ sbx::req (harness → worker) Redis Streams - │ sbx::res (worker → harness) - ▼ - @sh/sandbox-worker (new; runs INSIDE the sandbox, dials OUT to Redis) - │ - └─ executes bash -c locally, streams frames back -``` - -### Component boundaries - -| Unit | Purpose | Depends on | -|------|---------|-----------| -| `SandboxTransport` (interface) | The exec seam Pi sees. No transport knowledge above it. | — | -| `KubectlTransport` | Local/in-cluster fast path. Rename of today's `kubectlExecInPod`. | kubectl | -| `RedisChannelTransport` | Harness side: turn one `exec()` into req/res frames + correlation. | `@sh/work-queue` primitives, redis | -| `@sh/sandbox-worker` | Sandbox side: claim req frames, run locally, stream res frames, honor abort. | redis, local shell | -| `select-sandbox` (existing) | Now returns a **transport**, not a pod name. Chooses local vs redis per lease. | pool/lease logic | - -### The interface (fixed core) - -```ts -interface SandboxTransport { - exec(command: string, opts?: { - stdin?: Buffer; - onData?: (chunk: Buffer) => void; - signal?: AbortSignal; - timeout?: number; // seconds - }): Promise<{ stdout: Buffer; exitCode: number | null }>; - close(): Promise; -} -``` - -This is today's `ExecInPod` capability surface, verbatim, plus `close()`. Pi depends on -all of it (streaming via `onData`; abort via `signal`; `timeout`; `stdin` for writes), -so the interface must preserve all of it. - -### Architectural invariants - -- **Single worker per sandbox.** The `sbx::req|res` streams are per-sandbox and - owned by exactly one worker. This preserves shell/cwd/env/filesystem ordering, which - the load-balanced leaf queue explicitly does not guarantee. `xAutoClaim` reclaim is - used **only for crash recovery of that one worker**, never to load-balance across - workers. -- **One exec in flight per sandbox.** Mirrors `persistent-exec.ts` today; bounds memory - and makes req order == execution order. -- **Everything above `select-sandbox` is transport-blind.** `run-leaf`, `run-turn`, - `converge` receive a `SandboxTransport` and never learn how bytes reach the sandbox. - `converge` (repo sync) already goes through exec, so it rides the transport for free. - -### What does NOT change - -The Pi orchestration loop, `run-turn`, the session backend (`RedisSessionBackend`), and -the leaf queue (`@sh/work-queue`). The change slots strictly *below* the current -`ExecInPod` call sites (`converge.ts`, `run-leaf.ts`, `run-turn.ts`, `select-sandbox.ts`). - -## 3. Wire protocol (`sbx::req` / `sbx::res`) - -Reuses `packages/k8s-sandbox/src/framing.ts` (`wrapCommand` / `FrameParser`, -`\x01`-marker + base64 body, exit via `${PIPESTATUS[0]}`) inside the worker: the worker -runs the existing persistent bash locally and Redis simply replaces the kubectl -stdin/stdout pipe. The one extension: the persistent channel is one-shot per command -(no `onData` streaming) — for a *remote* sandbox we want live output, so the protocol -adds incremental `chunk` frames. - -**Correlation & ordering.** Each `exec()` gets a monotonic `reqId` (like -`persistent-exec`'s `seq`). One exec in flight ⇒ req order == execution order. `reqId` -doubles as the idempotency key. - -**Frames** (JSON envelopes; byte payloads base64-encoded): - -| Direction | `kind` | Fields | Purpose | -|-----------|--------|--------|---------| -| harness → `req` | `exec` | `reqId, command, stdinB64?, timeout, streaming` | run one command | -| harness → `req` | `abort` | `reqId` | cancel the in-flight exec | -| worker → `res` | `chunk` | `reqId, dataB64` | streamed stdout (0..n; only when `streaming`) | -| worker → `res` | `end` | `reqId, exitCode` | terminal success | -| worker → `res` | `error` | `reqId, message` | channel/exec failure → harness fallback or reject | - -- **Non-streaming ops** (read/write) emit a single `end` carrying full stdout — the - current one-shot frame. -- **Streaming ops** (bash/grep) emit `chunk`* then `end`; the harness replays each - `chunk` into `opts.onData`, matching today's `ExecInPod` contract byte-for-byte. - -**Abort & timeout.** The worker's req-read loop runs independently of the in-flight -child, so an `abort` frame is seen mid-exec → local `SIGKILL` (same effect as today's -`signal` → `child.kill`). Timeout is enforced **on both ends**: the worker kills locally -at `timeout`; the harness has its own deadline and synthesizes a timeout `error` if no -`end` arrives — it never hangs on a silent or malicious worker. - -**At-least-once → dedup.** The `req` stream uses a consumer group with **one consumer -(the worker)**; `xAutoClaim` reclaims an unacked req only for worker crash recovery. -Because a redelivered write is not idempotent, the worker keeps a bounded last-N -`reqId → end` cache and **re-emits the cached result instead of re-running**. -*Honest limitation:* if the worker died mid-write, exactly-once is impossible — the -contract is at-least-once + dedup-by-`reqId`, and partial filesystem effects on crash -are possible (same risk class as a leaf re-run today). - -**Poisoned-output defense (untrusted return path).** The harness enforces a -**total-output cap per exec**; on exceed it sends `abort`, truncates, and surfaces an -`[output truncated]` marker to Pi. This is the concrete mitigation for the driver-#1 -threat of a malicious sandbox flooding the model's context. - -## 4. The sandbox worker & transport selection - -### `@sh/sandbox-worker` (new; runs inside the sandbox, dials out) - -A thin, single-purpose process — a pump between Redis and one local persistent bash. -It holds **no LLM key and no orchestration**; it only executes commands and returns -bytes (the property that made "central brain" trust-correct). - -1. On start: read `SANDBOX_ID` + `REDIS_URL` from env; `ensureGroup()` on - `sbx::req`; connect; write + heartbeat a registration record. -2. Loop: `claim` next req frame (blocking read + `xAutoClaim` reclaim for - self-recovery). -3. `exec` frame → feed `wrapCommand(reqId, command, stdin)` into the local persistent - bash (reuse `framing.ts`); pipe the framed stdout back out as `chunk`/`end` frames on - `sbx::res`; `ack` after `end`. -4. `abort` frame → `SIGKILL` the local child. -5. Maintain the bounded `reqId → end` dedup cache; re-emit on redelivery. - -### Transport selection (`select-sandbox`, modified) - -`select-sandbox` already leases a sandbox from the pool via Redis. It now returns a -`SandboxTransport` instead of a pod name/config: - -``` -lease sandbox → inspect its registration record → - kind: "kubectl" → KubectlTransport(config) (local, fast — unchanged path) - kind: "redis" → RedisChannelTransport(sandboxId) (remote worker) -``` - -The pool's lease record gains a `transport: "kubectl" | "redis"` field (and, for redis, -the `sandboxId`). - -### Registration (v1: trusted-unreachable) - -A remote worker, on startup, writes a registration record to Redis -(`sandbox:registry:` with `transport:"redis"`, capacity, labels) and heartbeats it. -The pool's lease logic considers registered remote sandboxes alongside in-cluster pods; -missed heartbeats de-register. **This registry is the seam where Approach B / SPIFFE -authenticated registration later slots in for untrusted BYO — same record, stronger -identity.** - -## 5. Error handling & lifecycle - -| Failure | Detection | Behavior | -|---------|-----------|----------| -| Remote worker crash mid-exec | req frame unacked; `xAutoClaim` after `minIdleMs` | Redelivered to restarted worker. Dedup cache re-emits if already completed; else re-run (at-least-once). Partial-write risk documented. | -| Worker never starts / gone | missed heartbeat; harness deadline with no `end` | Lease marked unhealthy, sandbox evicted from pool; in-flight `exec` rejects with timeout `error`. Leaf retry re-leases a healthy sandbox. | -| Redis unreachable (harness) | connect/command error | `RedisChannelTransport.exec` rejects; leaf fails and re-queues via existing leaf `WorkQueue`. Redis is a documented availability dependency for remote execs. | -| Redis unreachable (worker) | connect error | Worker exits non-zero; supervisor restarts it; registry heartbeat lapses meanwhile. | -| In-flight exec exceeds timeout | dual deadline (§3) | Worker SIGKILLs local child; harness synthesizes timeout — never hangs on a silent worker. | -| Output cap exceeded (poisoned/runaway) | harness byte counter on `res` | Harness sends `abort`, truncates, surfaces `[output truncated]` to Pi. | -| Abort races with `end` | `reqId` correlation | Late `end` for an aborted `reqId` is dropped; abort for an already-ended `reqId` is a no-op. | -| Stale frames from a prior lease | `reqId`/session scoping | Worker resets stream position on (re)registration; harness ignores frames whose `reqId` predates the current transport instance. | - -**Lifecycle.** The transport is created at lease time; `close()` on leaf completion -(dispose local bash / stop reading `res`). The worker's lifetime is independent — it -outlives individual leases and serves whatever the pool assigns, matching the existing -shared-pool model. - -**Backpressure.** One exec in flight per sandbox (natural from a single persistent bash) -bounds memory; `chunk` size is capped so a single `res` entry stays small. - -## 6. Testing - -| Layer | What | How | -|-------|------|-----| -| Pure unit | Frame protocol, dedup cache, output-cap counter, timeout math, transport-selection logic | Plain vitest, no I/O. Extends existing `framing.ts` tests. | -| Contract | `RedisChannelTransport` + `@sh/sandbox-worker` against a real Redis, worker pointed at a local bash | Round-trip every op (read/write/bash/grep), abort mid-stream, timeout, redelivery→dedup, output-cap. Core suite. | -| Conformance | The *same* battery run against **both** `KubectlTransport` and `RedisChannelTransport` | One shared spec proving identical `SandboxTransport` contract — this is what makes them safely swappable (driver #2). | -| Live gate | Real worker pod dialing out to in-cluster Redis; one leaf end-to-end on Kind, then OCP | Follows the existing leaf-smoke pattern; manifest-shape vitest parses the worker Deployment YAML directly (no kustomize in CI). | - -## 7. Rollout & scope - -**Flagged, off by default, sequenced by trust:** - -1. `SandboxTransport` interface + `KubectlTransport` rename. **Pure refactor, zero - behavior change** — ships first, provably inert (conformance suite green, existing - e2e green). -2. `@sh/sandbox-worker` + `RedisChannelTransport` + registry, behind - `SH_REMOTE_SANDBOX=off` by default. No effect until a remote worker registers. -3. Enable on Kind/OCP for **trusted-unreachable** workers (Redis ACL per `sbx::*` - prefix + heartbeat registration). -4. **Deferred (explicit non-goal for this spec):** untrusted BYO isolation → Approach B - (SPIFFE-authenticated registration + per-connection identity via SPIRE/AuthBridge), - slotting into the same registry seam. - -**Scope boundaries (YAGNI):** no multi-worker-per-sandbox, no cross-region Redis, no -reverse-tunnel / gRPC (Approach B), no changes to the Pi loop or leaf queue. This spec -is exactly: the interface + one remote transport + the worker + the registry. - -## 8. Deferred: Approach B (for untrusted BYO) - -Recorded so the sequencing is explicit, not lost. When untrusted 3rd-party sandboxes -(driver #1) become real, Redis-as-shared-multi-tenant-bus is the weak point (coarse -ACLs; a scoping bug = cross-tenant exec injection). Approach B — the sandbox holds a -persistent outbound gRPC/WebSocket stream to a harness-side relay, with **per-connection -identity via mTLS/SPIFFE** (infra already operated via SPIRE + AuthBridge) — answers the -trust concern directly and gives protocol-native streaming/backpressure/abort. It plugs -into the **same registration seam** defined in §4, behind the **same `SandboxTransport` -interface** from §2, so adopting it is additive, not a rewrite. From 77d19d1adc9f18ad0bf8719838d1ce5e014bf4ed Mon Sep 17 00:00:00 2001 From: Paolo Dettori Date: Thu, 9 Jul 2026 11:44:41 -0400 Subject: [PATCH 3/3] docs: Add ADR-0024 for SandboxTransport remote exec design Record the language-neutral gRPC SandboxTransport decision from the 2026-07-08 spec as ADR-0024 (Accepted), and add its index row to the ADR README. Assisted-By: Claude (Anthropic AI) Signed-off-by: Paolo Dettori --- .../0024-sandbox-transport-remote-exec.md | 30 +++++++++++++++++++ docs/adrs/README.md | 1 + 2 files changed, 31 insertions(+) create mode 100644 docs/adrs/0024-sandbox-transport-remote-exec.md diff --git a/docs/adrs/0024-sandbox-transport-remote-exec.md b/docs/adrs/0024-sandbox-transport-remote-exec.md new file mode 100644 index 0000000..ff69a31 --- /dev/null +++ b/docs/adrs/0024-sandbox-transport-remote-exec.md @@ -0,0 +1,30 @@ +# ADR-0024: Remote sandbox exec over a worker-dialed gRPC stream, contract as language-neutral Protobuf + +- **Status:** Accepted +- **Date:** 2026-07-08 +- **Deciders:** Serverless Harness team +- **Spec:** [`../specs/2026-07-08-sandbox-transport-grpc-design.md`](../specs/2026-07-08-sandbox-transport-grpc-design.md) + +## Context + +The harness runs every Pi tool call inside a sandbox pod via `kubectl exec`, so it must dial *into* the pod through the kube API. That rules out any sandbox behind NAT, on-prem, on a laptop, or in another cloud — and blocks the top driver, bring-your-own (untrusted third-party) sandboxes. Reaching those requires inverting connectivity (the sandbox dials *out*) with a contract that is language-neutral (any runtime can host a worker) and firewall-friendly (one outbound TLS connection on `:443`), without touching the Pi loop, the session backend, or the leaf queue. An earlier revision of this PR got the outbound-dial direction right but carried the RPC over Redis Streams behind a TypeScript interface — locking workers to TS via JSON+base64 frames and forcing Redis (a port `:443`-only egress commonly blocks) into the exec path. + +## Decision + +We will delegate remote command execution over a single **worker-dialed gRPC bidirectional `Attach` stream (HTTP/2 on `:443`)**, define the contract as a **Protobuf IDL (`sandbox/v1`)** rather than a language-specific interface, and land both paths behind the existing **`SandboxTransport`** seam — `KubectlTransport` (today's in-cluster fast path, renamed) and a new `GrpcRelayTransport` are two implementations of one interface. A **single-replica, presence-only** in-cluster relay bridges the worker's outbound stream to the harness's in-cluster `SandboxExec` calls and mirrors connected workers into the existing Redis sandbox pool; matching stays in `select-sandbox`. One **Go reference worker** ships as the honest proof the contract is genuinely language-neutral. + +### Alternatives considered + +- **Redis-Streams transport + TS `@sh/sandbox-worker`** (this PR's prior revision) — rejected: TS lock-in through JSON+base64 frames, and Redis-on-its-port is blocked by `:443`-only egress while forcing a Redis dependency into exec. +- **Connect / HTTP-1.1 fallback** — rejected: full-duplex bidi needs HTTP/2 regardless, so Connect adds a second toolchain for no gain on the streaming core. +- **Relay owning matching / multi-replica HA** — deferred: a single replica needs no presence glue beyond the pool mirror, and matching stays in the existing pool. + +## Consequences + +- Positive: a sandbox can live anywhere behind one outbound `:443` connection with no inbound rules; any gRPC-capable language can host a worker; the in-cluster `kubectl-exec` path is unchanged; everything above `select-sandbox` stays transport-blind; the frame semantics (`req_id` correlation, at-least-once + dedup, dual-ended timeout, per-exec output cap) carry over verbatim. +- Negative / accepted cost: a new in-cluster relay component plus a Protobuf/gRPC toolchain; single-replica relay means a restart drops all parked streams and fails in-flight execs (recovery is leaf retry — no mid-exec durability); exactly-once is impossible — at-least-once + dedup only, with partial-write risk on crash. +- Follow-up owed: untrusted-BYO SPIFFE/mTLS on the same `Attach` endpoint; multi-replica relay HA; private-mesh reachability (Headscale / WireGuard); additional-language workers; HTTP/1.1-only proxy traversal — all additive behind the same seam. + +--- + +*Assisted-By: Claude (Anthropic AI) * diff --git a/docs/adrs/README.md b/docs/adrs/README.md index 4af4a66..b5e5642 100644 --- a/docs/adrs/README.md +++ b/docs/adrs/README.md @@ -34,6 +34,7 @@ spec). Chronological by the spec's date; numbers are permanent. | [0021](0021-shared-sandbox-pool.md) | Shared sandbox pool via N Sandbox CRs with harness-side Redis-lease routing | Implemented | | [0022](0022-workload-parameterized-sandbox-load.md) | Report the sharing ratio as an N-vs-workload curve, not a single number | Implemented | | [0023](0023-sandbox-sharing-ratio-experiments.md) | Measure sandbox sharing capacity on runc; split Kata isolation into P4 | Implemented | +| [0024](0024-sandbox-transport-remote-exec.md) | Remote sandbox exec over a worker-dialed gRPC stream, contract as language-neutral Protobuf | Accepted | ## What an ADR is (and isn't)