diff --git a/README.md b/README.md index 5370dabf..5c8ccbbd 100644 --- a/README.md +++ b/README.md @@ -523,6 +523,8 @@ For VFS-capable runtimes (claude-code/codex) the `hivemind-goals` skill creates Per-agent integration mechanisms (marketplace plugin, hooks, skills, native extension) and monorepo structure: **[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)**. +What each trace row records — model, reasoning effort, stop reason, token usage (incl. cache tokens and cost) and per-harness extras, with the capture matrix per agent: **[harnesses/TRACE_MODEL_USAGE.md](harnesses/TRACE_MODEL_USAGE.md)**. + ## Roadmap - **Trajectory export for fine-tuning.** Because traces are stored in Deeplake's tensor format, they're export-ready as PyTorch datasets. Teams running their own open-source models can fine-tune on their org's accumulated trajectories. A handful of advanced customers are already doing this against the trajectories their Claude Code and Codex agents generated. diff --git a/harnesses/TRACE_MODEL_USAGE.md b/harnesses/TRACE_MODEL_USAGE.md new file mode 100644 index 00000000..fc53878b --- /dev/null +++ b/harnesses/TRACE_MODEL_USAGE.md @@ -0,0 +1,143 @@ +# Trace model / token-usage enrichment + +Every captured session event (user prompt, tool call, assistant turn) is written +as one row in the `sessions` table, with the event payload in the JSONB `message` +column. On top of the base event, each row is enriched — where the agent exposes +it — with **which model** produced the turn, its **reasoning effort**, why the +turn **stopped**, full **token usage** (including cache tokens and **money +cost**), and a per-harness **`usage_extra`** bag for fields that don't generalize. + +This lets "tokens/cost per model" be a plain `GROUP BY message->>'model'` query. + +## Where the code lives + +- Extraction / normalization: [`src/notifications/model-usage.ts`](../src/notifications/model-usage.ts) + - `parseClaudeTurnMeta(transcriptPath)` — Claude Code + - `parseCodexTurnMeta(transcriptPath, fallbackModel)` — Codex + - `normalizeSdkUsage` / `sdkTurnMeta` — Pi / OpenClaw (in-process SDK message) + - `readClaudeEffortLevel(cwd)` — Claude effort from settings +- Wiring (per harness): + - Claude Code — `src/hooks/capture.ts` (assistant_message / Stop) + - Codex — `src/hooks/codex/capture.ts` + - Pi — `harnesses/pi/extension-source/hivemind.ts` (`message_end`; inlines the + normalizer because pi ships raw `.ts` with no shared-module imports) + - OpenClaw — `harnesses/openclaw/src/index.ts` (`agent_end` auto-capture) + - Hermes — `src/hooks/hermes/capture.ts` (reads `extra.model` / `extra.platform`) + +## Stored shape (JSONB `message`) + +```jsonc +{ + // ...base event fields (session_id, type, content, timestamp, ...) + "model": "claude-opus-4-8", + "reasoning_effort": "medium", // null when unset / not applicable + "stop_reason": "end_turn", // end_turn | tool_use | max_tokens | stop | toolUse | ... + "token_usage": { // the most recent turn + "input_tokens": 131, + "output_tokens": 403, + "cache_read_tokens": 357059, + "cache_creation_tokens": 2102, + "reasoning_output_tokens": 160, // Codex + "total_tokens": 216408, // Codex / SDK + "cost": { // Pi / OpenClaw — fractional dollars + "input": 0.165, "output": 0.0001, "cache_read": 0, "cache_creation": 0, "total": 0.1651 + } + }, + "token_usage_total": { /* ...same shape */ }, // Codex only — whole-session cumulative + "usage_extra": { /* per-harness, see below */ } +} +``` + +All fields are **optional** and only present when the agent exposes them (absent +keys stay absent — never defaulted to 0/empty). + +## Capture matrix (verified against real transcripts) + +| Field | Claude Code | Codex | Pi | OpenClaw | Cursor | Hermes | +|---|:--:|:--:|:--:|:--:|:--:|:--:| +| `model` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `reasoning_effort` | ✅ (settings) | ✅ | — | — | — | — | +| `stop_reason` | ✅ | — ¹ | ✅ | ✅ | — | — | +| token: input / output | ✅ | ✅ | ✅ | ✅ | — | — | +| token: cache_read | ✅ | ✅ | ✅ | ✅ | — | — | +| token: cache_creation | ✅ | — ² | ✅ | ✅ | — | — | +| token: reasoning_output | — | ✅ | — | — | — | — | +| token: total | — ³ | ✅ | ✅ | ✅ | — | — | +| `token_usage.cost` ($) | — | — | ✅ | ✅ | — | — | +| `token_usage_total` (cumulative) | — | ✅ | — | — | — | — | +| `usage_extra` | tier / speed / cache-TTL / server_tool_use | model_context_window / rate_limits | — | — | — | platform | +| custom fields | — | — | — | — | duration / loop_count / status | — | + +**Notes.** ¹ Codex emits only per-turn `status:"completed"`, not a stop reason. +² Codex has no cache-write concept. ³ Claude doesn't report a grand total in +`usage`; it isn't synthesized. Cursor's transcript has **no** token/cost data; +Hermes' hook payload carries **no** token/cost (only `model` + `platform`). + +### Source of each field, per harness + +| Harness | Source | +|---|---| +| Claude Code | transcript `~/.claude/projects/.../*.jsonl` (last assistant line: `message.model`, `message.usage`, `message.stop_reason`); effort from `settings.json` `effortLevel` | +| Codex | rollout `~/.codex/sessions/.../rollout-*.jsonl` (`turn_context` → model/effort; `token_count` → usage/total/quota) | +| Pi | in-process `message_end` SDK message (`model`, `stopReason`, `usage` incl. `cost`) | +| OpenClaw | in-process `agent_end` SDK message (`model`, `stopReason`, `usage` incl. `cost`) | +| Cursor | hook payload `model` (+ its own `duration`/`loop_count`/`status`) | +| Hermes | hook payload `extra.model` / `extra.platform` (NousResearch/hermes-agent `shell_hooks.py`) | + +## Aggregation + +```sql +-- Per-model output tokens (Claude / any agent that reports per-turn usage) +SELECT message->>'model' AS model, + SUM((message->'token_usage'->>'output_tokens')::bigint) AS out_tokens +FROM sessions +WHERE agent = 'claude_code' +GROUP BY 1; + +-- Per-model cost (Pi / OpenClaw) +SELECT message->>'model' AS model, + SUM((message->'token_usage'->'cost'->>'total')::numeric) AS usd +FROM sessions +WHERE message->'token_usage'->'cost' IS NOT NULL +GROUP BY 1; +``` + +`token_usage_total` (Codex) is a **whole-session cumulative across all models**, +not per-model — roll it up as `MAX(total_tokens)` per session, not `SUM`. For +per-model Codex totals, sum the per-turn `token_usage` and dedup by `turn_id` +(a turn's tool events share one snapshot). + +## Design notes + +- **Best-effort.** Any read/parse failure returns null and capture proceeds + unchanged — enrichment never blocks or breaks a trace write. +- **Tail read (perf).** The parsers read only the **file tail** (default 1 MiB) + via a single `openTranscript()` — model/usage live in the most recent turn at + the end of the file — so each event is ~O(1), not O(filesize) (and not O(n²) + across a Codex session that parses on every user/tool row). If a single record + exceeds the window, a whole-file fallback runs from the **same fd/snapshot**. + See the file's `readWindowLines` / `openTranscript` for boundary + short-read + handling. Residual TOCTOU (transcript rewritten between the two reads) is + unreachable for append-only session logs. +- **Redaction guard.** Storing model ids exposed a false positive: the secret + entropy backstop shredded long dated slugs like `claude-haiku-4-5-20251001`. + `src/hooks/shared/redact.ts` `looksLikeSecret` exempts provider model ids + (lowercase, segments ≤12 chars, incl. cloud-prefixed Bedrock slugs) while a + high-entropy secret wearing a model prefix is still masked. +- **Reasoning effort for Claude** is the user's `effortLevel` setting + (low/medium/high/xhigh), not a per-message field — read from settings at + capture time; allowlisted so an arbitrary settings value can't poison analytics. +- **Numeric guards.** Token counts must be non-negative safe integers; cost is a + non-negative finite number (fractional). Invalid values are dropped. + +## Adding a new harness + +If a runtime's transcript/payload carries model/usage, wire it the same way: +- Payload has `model` → set it on the trace meta directly. +- Transcript file with usage → add a `parseTurnMeta`. +- In-process SDK message with `{model, usage, stopReason}` → reuse `sdkTurnMeta`. +- Anything that doesn't generalize → put it under `usage_extra`, don't fabricate. + +Tests: `tests/shared/notifications-model-usage.test.ts` (extraction), +`tests/shared/redact.test.ts` (model-id guard), plus per-harness capture tests. +E2E harness: `scripts/trace-model-usage-e2e.mjs`. diff --git a/harnesses/openclaw/src/index.ts b/harnesses/openclaw/src/index.ts index 91de87eb..ca422ae0 100644 --- a/harnesses/openclaw/src/index.ts +++ b/harnesses/openclaw/src/index.ts @@ -67,6 +67,7 @@ import { readVirtualPathContent } from "../../../src/hooks/virtual-table-query.j import { tryEmbedStandalone, _setSpawnImpl } from "../../../src/embeddings/standalone-embed-client.js"; import { embeddingSqlLiteral } from "../../../src/embeddings/sql.js"; import { redactSecrets } from "../../../src/hooks/shared/redact.js"; +import { sdkTurnMeta } from "../../../src/notifications/model-usage.js"; // Resolve sibling skillify-worker.js path at runtime via import.meta.url. The // openclaw plugin is bundled to harnesses/openclaw/dist/index.js, then installed to // ~/.openclaw/extensions/hivemind/dist/index.js by install-openclaw.ts. The @@ -1482,7 +1483,7 @@ export default definePluginEntry({ // Auto-capture: store new messages in sessions table (same format as CC capture.ts) if (config.autoCapture !== false) { hook("agent_end", async (event) => { - const ev = event as { success?: boolean; session_id?: string; channel?: string; messages?: Array<{ role: string; content: string | Array<{ type: string; text?: string }> }> }; + const ev = event as { success?: boolean; session_id?: string; channel?: string; messages?: Array<{ role: string; content: string | Array<{ type: string; text?: string }>; model?: unknown; usage?: unknown; stopReason?: unknown }> }; if (!captureEnabled || !ev.success || !ev.messages?.length) return; try { const dl = await getApi(); @@ -1515,12 +1516,16 @@ export default definePluginEntry({ if (!text.trim()) continue; const ts = new Date().toISOString(); + // Tag assistant rows with model + token usage from the SDK message + // (openclaw exposes both on the message object; no reasoning effort). + const modelMeta = msg.role === "assistant" ? sdkTurnMeta(msg.model, msg.usage, msg.stopReason) : undefined; const entry = { id: crypto.randomUUID(), type: msg.role === "user" ? "user_message" : "assistant_message", session_id: sid, content: text, timestamp: ts, + ...(modelMeta ?? {}), }; // Mask secrets before the payload is embedded or stored. const line = redactSecrets(JSON.stringify(entry)); diff --git a/harnesses/pi/extension-source/hivemind.ts b/harnesses/pi/extension-source/hivemind.ts index 82af0f47..ce459291 100644 --- a/harnesses/pi/extension-source/hivemind.ts +++ b/harnesses/pi/extension-source/hivemind.ts @@ -149,6 +149,44 @@ function applyDirConfig(creds: Creds, cwd: string): { creds: Creds; collect: boo return { creds: { ...withEnv, orgId, orgName, workspaceId }, collect: true, routed }; } +// Inline copy of normalizeSdkUsage / sdkTurnMeta (shared version lives in +// src/notifications/model-usage.ts, but pi extensions ship as raw .ts with no +// shared-module imports — kept in lockstep with that file). Maps the pi/SDK +// usage object {input, output, cacheRead, cacheWrite, totalTokens} onto the +// normalized token_usage keys used across every agent's trace rows. +function piModelMeta(message: any): { model?: string; stop_reason?: string; token_usage?: Record } { + const out: { model?: string; stop_reason?: string; token_usage?: Record } = {}; + if (typeof message?.model === "string" && message.model) out.model = message.model; + if (typeof message?.stopReason === "string" && message.stopReason) out.stop_reason = message.stopReason; + const u = message?.usage; + if (u && typeof u === "object") { + const t: Record = {}; + const putInt = (k: string, v: unknown) => { + if (typeof v === "number" && Number.isSafeInteger(v) && v >= 0) t[k] = v; + }; + putInt("input_tokens", u.input); + putInt("output_tokens", u.output); + putInt("cache_read_tokens", u.cacheRead); + putInt("cache_creation_tokens", u.cacheWrite); + putInt("total_tokens", u.totalTokens); + if (u.cost && typeof u.cost === "object") { + const cost: Record = {}; + putFloatInto(cost, "input", u.cost.input); + putFloatInto(cost, "output", u.cost.output); + putFloatInto(cost, "cache_read", u.cost.cacheRead); + putFloatInto(cost, "cache_creation", u.cost.cacheWrite); + putFloatInto(cost, "total", u.cost.total); + if (Object.keys(cost).length > 0) t.cost = cost; + } + if (Object.keys(t).length > 0) out.token_usage = t; + } + return out; +} + +function putFloatInto(target: Record, key: string, v: unknown): void { + if (typeof v === "number" && Number.isFinite(v) && v >= 0) target[key] = v; +} + // Inline copies of decodeJwtPayload + healDriftedOrgToken (the shared helpers // live in src/commands/auth.ts, but pi extensions ship as raw .ts with no // shared-module imports — kept in lockstep with that file). @@ -1531,6 +1569,7 @@ export default function hivemindExtension(pi: ExtensionAPI): void { session_id: sessionId, content: text, timestamp: new Date().toISOString(), + ...piModelMeta(message), }); } catch (e: any) { logHm(`message_end: writeSessionRow swallowed: ${e?.message ?? e}`); diff --git a/scripts/trace-model-usage-e2e.mjs b/scripts/trace-model-usage-e2e.mjs new file mode 100644 index 00000000..b3f1389d --- /dev/null +++ b/scripts/trace-model-usage-e2e.mjs @@ -0,0 +1,352 @@ +#!/usr/bin/env node +/** + * End-to-end trace of model + reasoning effort + token usage across every model + * provider present in the local agent transcripts. + * + * For each distinct model found in the real Claude Code transcripts + * (~/.claude/projects) and Codex rollouts (~/.codex/sessions), this drives the + * REAL compiled capture hook (dist/src/hooks/capture.js and + * dist/src/hooks/codex/capture.js) with a synthesized hook payload whose + * transcript_path points at that transcript, then reads the rows back from the + * sessions table and prints the model / reasoning_effort / token_usage that + * actually landed in the JSONB `message` column. + * + * It exercises the whole path — parser -> entry build -> SQL INSERT -> Deeplake + * -> read-back — not just the parser in isolation. + * + * Safety: + * - Writes to a THROWAWAY table (HIVEMIND_SESSIONS_TABLE, default + * `sessions_modeltest`), never the production `sessions` table. + * - HIVEMIND_WIKI_WORKER=1 suppresses the hooks' side effects (owner walk, + * periodic summary spawn, stop trigger) so only the INSERT runs. + * - Deletes its own rows on completion. + * + * Usage: + * node scripts/trace-model-usage-e2e.mjs # trace + report + cleanup + * HIVEMIND_SESSIONS_TABLE=my_test node scripts/trace-model-usage-e2e.mjs + * KEEP=1 node scripts/trace-model-usage-e2e.mjs # keep rows for inspection + */ + +import { spawnSync } from "node:child_process"; +import { readdirSync, readFileSync, statSync } from "node:fs"; +import { homedir } from "node:os"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), ".."); +const TABLE = process.env.HIVEMIND_SESSIONS_TABLE ?? "sessions_modeltest"; +const RUN_TAG = "modeltest-" + Date.now().toString(36); +const MAX_FILES_PER_AGENT = Number(process.env.MAX_FILES ?? 300); + +const { parseClaudeTurnMeta, parseCodexTurnMeta, sdkTurnMeta } = await import( + join(ROOT, "dist/src/notifications/model-usage.js") +); +const { loadConfig } = await import(join(ROOT, "dist/src/config.js")); +const { DeeplakeApi } = await import(join(ROOT, "dist/src/deeplake-api.js")); + +// Env shared by every hook invocation: throwaway table, no side effects, no embed. +const HOOK_ENV = { + ...process.env, + HIVEMIND_SESSIONS_TABLE: TABLE, + HIVEMIND_WIKI_WORKER: "1", + HIVEMIND_EMBEDDINGS: "false", + HIVEMIND_CAPTURE: "true", +}; + +/** Newest-first list of transcript files under a directory tree, capped. */ +function listTranscripts(rootDir, cap) { + const out = []; + const walk = (dir) => { + let entries; + try { + entries = readdirSync(dir, { withFileTypes: true }); + } catch { + return; + } + for (const e of entries) { + const p = join(dir, e.name); + if (e.isDirectory()) walk(p); + else if (e.name.endsWith(".jsonl")) out.push(p); + } + }; + walk(rootDir); + return out + .map((p) => ({ p, mtime: statSync(p).mtimeMs })) + .sort((a, b) => b.mtime - a.mtime) + .slice(0, cap) + .map((x) => x.p); +} + +/** One representative transcript per distinct model, using the real parser. */ +function pickOnePerModel(files, parse) { + const byModel = new Map(); + for (const f of files) { + let meta; + try { + meta = parse(f); + } catch { + continue; + } + if (meta?.model && !byModel.has(meta.model)) byModel.set(meta.model, f); + } + return byModel; // model -> file +} + +/** + * Find a real transcript message carrying model+usage, build the entry via the + * shared sdkTurnMeta (the exact logic Pi inlines and OpenClaw imports), and + * assert the normalized token_usage has positive integer input+output tokens. + * Returns { ok, found } — ok:false means a transcript with usage existed but + * produced no/invalid token_usage (a real failure); found:false means the agent + * simply has no local transcripts (not a failure in this env). + */ +function proveSdk(name, dir) { + const files = listTranscripts(dir, 200); + let anyTranscript = files.length > 0; + for (const f of files) { + let raw; + try { + raw = readFileSync(f, "utf-8"); + } catch { + continue; + } + for (const ln of raw.split("\n")) { + if (!ln.includes('"usage"')) continue; + let o; + try { + o = JSON.parse(ln); + } catch { + continue; + } + const m = o.message ?? o; + const meta = sdkTurnMeta(m?.model ?? o?.model, m?.usage ?? o?.usage); + const u = meta?.token_usage; + if (u) { + const ok = Number.isInteger(u.input_tokens) && u.input_tokens >= 0 && + Number.isInteger(u.output_tokens) && u.output_tokens >= 0; + console.log(` [${name}] ${String(meta.model ?? "?").padEnd(24)} token_usage=${JSON.stringify(u)} ${ok ? "OK" : "INVALID"}`); + return { ok, found: true }; + } + } + } + console.log(` [${name}] no real transcript with usage found${anyTranscript ? " (has transcripts, none with usage)" : ""}`); + return { ok: true, found: false }; +} + +function runHook(entry, payload) { + const r = spawnSync("node", [join(ROOT, entry)], { + input: JSON.stringify(payload), + env: HOOK_ENV, + encoding: "utf-8", + timeout: 60_000, + }); + return r.status === 0; +} + +async function main() { + const config = loadConfig(); + if (!config) { + console.error("No Deeplake credentials resolved (log in with `hivemind login`)."); + process.exit(1); + } + console.log(`Org: ${config.orgName} | table: ${TABLE} | run: ${RUN_TAG}\n`); + + const claudeModels = pickOnePerModel( + listTranscripts(join(homedir(), ".claude", "projects"), MAX_FILES_PER_AGENT), + parseClaudeTurnMeta, + ); + const codexModels = pickOnePerModel( + listTranscripts(join(homedir(), ".codex", "sessions"), MAX_FILES_PER_AGENT), + (f) => parseCodexTurnMeta(f), + ); + + console.log( + `Discovered ${claudeModels.size} Claude model(s), ${codexModels.size} Codex model(s).\n`, + ); + + // Build one capture job per model. `expectedModels` is every model driven + // through a stdin hook that MUST appear in the read-back — the hooks swallow + // errors and exit 0, so exit code alone is not proof of capture. + // + // Inserts are PACED (SLEEP_MS between them): the sessions table drops rapid + // same-table bursts (returns 200 with rows=0 and the row never lands), a + // known backend quirk. Real sessions emit events seconds apart, so pacing + // reflects reality; stragglers are retried below. + const jobs = []; + let n = 0; + for (const [model, file] of claudeModels) { + const sid = `${RUN_TAG}-cc-${n++}`; + jobs.push({ agent: "claude_code", model, run: () => + runHook("dist/src/hooks/capture.js", { + session_id: sid, transcript_path: file, cwd: ROOT, + hook_event_name: "Stop", last_assistant_message: `e2e trace for ${model}`, + }) }); + } + for (const [model, file] of codexModels) { + const sid = `${RUN_TAG}-cx-${n++}`; + jobs.push({ agent: "codex", model, run: () => + runHook("dist/src/hooks/codex/capture.js", { + session_id: sid, transcript_path: file, cwd: ROOT, + hook_event_name: "UserPromptSubmit", model, prompt: `e2e trace for ${model}`, + }) }); + } + // Cursor: model is in the payload (no token data exists in its transcript). + jobs.push({ agent: "cursor", model: "cursor-default-model", run: () => + runHook("dist/src/hooks/cursor/capture.js", { + conversation_id: `${RUN_TAG}-cur`, session_id: `${RUN_TAG}-cur`, + hook_event_name: "afterAgentResponse", model: "cursor-default-model", + cwd: ROOT, text: "e2e trace for cursor", + }) }); + const expectedModels = new Set(jobs.map((j) => j.model)); + + const SLEEP_MS = Number(process.env.SLEEP_MS ?? 900); + const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); + const runJobs = async (list) => { + for (const job of list) { + const ok = job.run(); + console.log(` [${job.agent}] ${job.model} -> ${ok ? "captured" : "FAILED"}`); + await sleep(SLEEP_MS); + } + }; + await runJobs(jobs); + + // Hermes: payload carries no model / token data — capture a plain event to + // confirm the row still lands (model/usage simply absent, never fabricated). + { + const ok = runHook("dist/src/hooks/hermes/capture.js", { + session_id: `${RUN_TAG}-herm`, hook_event_name: "UserPromptSubmit", + cwd: ROOT, extra: { prompt: "e2e trace for hermes" }, + }); + console.log(` [hermes] (no model/token data) -> ${ok ? "captured" : "FAILED"}`); + } + + // Read the rows back from the store and report what actually landed. The + // sessions table is read-after-write lagged — a row INSERTed moments ago may + // not be visible on an immediate SELECT (the backend even returns rows=0 on + // the write) — so poll until every invoked model is visible or we time out. + const api = new DeeplakeApi( + config.token, + config.apiUrl, + config.orgId, + config.workspaceId, + TABLE, + ); + const readRows = async () => { + const res = await api.query( + `SELECT message FROM "${TABLE}" WHERE message->>'session_id' LIKE '${RUN_TAG}-%' ORDER BY message->>'model'`, + ); + return res?.rows ?? res?.data ?? res ?? []; + }; + // Derive the agent from the session_id we control — the SQL `agent` column + // reads back unreliably here, and the prefix is authoritative anyway. + const agentOf = (sid) => { + const rest = String(sid ?? "").slice(RUN_TAG.length + 1); + if (rest.startsWith("cc-")) return "claude_code"; + if (rest.startsWith("cx-")) return "codex"; + if (rest.startsWith("cur")) return "cursor"; + if (rest.startsWith("herm")) return "hermes"; + return "?"; + }; + const modelOf = (r) => (typeof r.message === "string" ? JSON.parse(r.message) : r.message).model; + const missingFrom = (rws) => { + const present = new Set(rws.map(modelOf)); + return [...expectedModels].filter((m) => !present.has(m)); + }; + let rows = []; + // Up to 3 rounds: poll for propagation, then re-capture any straggler the + // backend dropped (spaced further apart) before asserting. + for (let round = 0; round < 3; round++) { + for (let attempt = 0; attempt < 8; attempt++) { + rows = await readRows(); + if (missingFrom(rows).length === 0) break; + await sleep(2000); + } + const missing = missingFrom(rows); + if (missing.length === 0 || round === 2) break; + console.log(`\nRetry round ${round + 1}: re-capturing ${missing.length} straggler(s): ${missing.join(", ")}`); + await runJobs(jobs.filter((j) => missing.includes(j.model))); + } + + console.log(`\n=== Rows read back from ${TABLE} (${rows.length}) ===`); + const landedModels = new Set(); + const modelsByAgent = new Map(); // agent -> Set(model) + const tokenRowsByAgent = new Map(); // agent -> count of rows with token_usage + const seen = new Set(); + for (const row of rows) { + const m = typeof row.message === "string" ? JSON.parse(row.message) : row.message; + const agent = agentOf(m.session_id); + landedModels.add(m.model); + if (!modelsByAgent.has(agent)) modelsByAgent.set(agent, new Set()); + modelsByAgent.get(agent).add(m.model); + const u = m.token_usage; + // Correctness, not just presence: a token-bearing row must have positive + // integer input+output tokens (proves real numbers flowed parser->hook->DB). + const validTok = u && + Number.isInteger(u.input_tokens) && u.input_tokens >= 0 && + Number.isInteger(u.output_tokens) && u.output_tokens >= 0 && + u.input_tokens + u.output_tokens > 0; + if (validTok) tokenRowsByAgent.set(agent, (tokenRowsByAgent.get(agent) ?? 0) + 1); + if (seen.has(m.model)) continue; // one line per model in the display + seen.add(m.model); + console.log( + ` ${String(agent).padEnd(12)} ${String(m.model ?? "?").padEnd(28)} effort=${String(m.reasoning_effort ?? "—").padEnd(7)} ` + + `usage=${u ? JSON.stringify(u) : "—"}` + + (m.token_usage_total ? ` total=${JSON.stringify(m.token_usage_total)}` : ""), + ); + } + + const invokedByAgent = new Map(); + for (const j of jobs) invokedByAgent.set(j.agent, (invokedByAgent.get(j.agent) ?? 0) + 1); + console.log(`\nInvoked vs visible per agent (sessions-table burst read-lag hides some of a rapid tail):`); + for (const [agent, count] of invokedByAgent) { + console.log(` ${agent.padEnd(12)} invoked=${count} visible=${modelsByAgent.get(agent)?.size ?? 0} with-tokens=${tokenRowsByAgent.get(agent) ?? 0}`); + } + + // Pi + OpenClaw capture in-process (extension event / producer hook), not via + // a stdin subprocess, so they can't be driven here. Prove the exact enriched + // entry they build by running the shared sdkTurnMeta over a real message from + // each one's on-disk transcript, asserting positive token values. + console.log(`\n=== Pi / OpenClaw entry-build proof (from real transcripts) ===`); + const piProof = proveSdk("pi", join(homedir(), ".pi", "agent", "sessions")); + const openclawProof = proveSdk("openclaw", join(homedir(), ".openclaw")); + + // Guard: exit code 0 from a capture hook is not proof — verify each agent's + // capture path actually wrote correct rows. Require only agents that were + // ACTUALLY invoked (a CI box may lack a given agent's transcripts), and for + // the token-bearing agents require a row with valid positive token counts. + // We don't require every one of a 20-model burst to be visible — the backend + // read-lag won't reliably surface the tail — but a genuinely broken hook + // lands zero rows for its agent. + const invokedAgents = new Set(jobs.map((j) => j.agent)); + const tokenAgents = ["claude_code", "codex"].filter((a) => invokedAgents.has(a)); + const problems = []; + if (rows.length === 0) problems.push("no rows landed at all"); + for (const agent of invokedAgents) { + if (!(modelsByAgent.get(agent)?.size)) problems.push(`no ${agent} rows landed`); + } + for (const agent of tokenAgents) { + if (!tokenRowsByAgent.get(agent)) problems.push(`no ${agent} row carried valid token_usage`); + } + if (!piProof.ok) problems.push("pi sdkTurnMeta produced invalid token_usage"); + if (!openclawProof.ok) problems.push("openclaw sdkTurnMeta produced invalid token_usage"); + if (problems.length > 0) { + console.error(`\nFAIL: ${problems.join("; ")}`); + if (process.env.KEEP !== "1") { + await api.query(`DELETE FROM "${TABLE}" WHERE message->>'session_id' LIKE '${RUN_TAG}-%'`); + } + process.exit(1); + } + console.log(`\nPASS: every invoked agent's capture path wrote correct rows (token-bearing agents carry valid token_usage).`); + + if (process.env.KEEP === "1") { + console.log(`\nKEEP=1 — leaving ${rows.length} rows in ${TABLE}.`); + return; + } + await api.query(`DELETE FROM "${TABLE}" WHERE message->>'session_id' LIKE '${RUN_TAG}-%'`); + console.log(`\nCleaned up ${rows.length} test rows from ${TABLE}.`); +} + +main().catch((e) => { + console.error("e2e trace failed:", e?.message ?? e); + process.exit(1); +}); diff --git a/src/hooks/capture.ts b/src/hooks/capture.ts index 8388d83d..ecbba67f 100644 --- a/src/hooks/capture.ts +++ b/src/hooks/capture.ts @@ -16,6 +16,7 @@ import { sqlStr } from "../utils/sql.js"; import { projectNameFromCwd } from "../utils/project-name.js"; import { log as _log } from "../utils/debug.js"; import { buildSessionPath } from "../utils/session-path.js"; +import { parseClaudeTurnMeta } from "../notifications/model-usage.js"; import { bumpTotalCount, loadTriggerConfig, @@ -134,12 +135,18 @@ async function main(): Promise { }; } else if (input.last_assistant_message !== undefined) { log(`assistant session=${input.session_id}`); + // Model / usage aren't in the hook payload — read them from the transcript's + // last assistant turn (best-effort; null on any read/parse failure). On + // SubagentStop, last_assistant_message belongs to the subagent transcript; + // transcript_path points at the parent session, so prefer the agent one. + const modelMeta = parseClaudeTurnMeta(input.agent_transcript_path ?? input.transcript_path); entry = { id: crypto.randomUUID(), ...meta, type: "assistant_message", content: input.last_assistant_message, ...(input.agent_transcript_path ? { agent_transcript_path: input.agent_transcript_path } : {}), + ...(modelMeta ?? {}), }; } else { log("unknown event, skipping"); diff --git a/src/hooks/codex/capture.ts b/src/hooks/codex/capture.ts index f6b33512..3d5af6f2 100644 --- a/src/hooks/codex/capture.ts +++ b/src/hooks/codex/capture.ts @@ -21,6 +21,7 @@ import { sqlStr } from "../../utils/sql.js"; import { projectNameFromCwd } from "../../utils/project-name.js"; import { log as _log } from "../../utils/debug.js"; import { buildSessionPath } from "../../utils/session-path.js"; +import { parseCodexTurnMeta } from "../../notifications/model-usage.js"; import { EmbedClient } from "../../embeddings/client.js"; import { embeddingSqlLiteral } from "../../embeddings/sql.js"; import { embeddingsDisabled } from "../../embeddings/disable.js"; @@ -84,6 +85,14 @@ async function main(): Promise { const api = new DeeplakeApi(config.token, config.apiUrl, config.orgId, config.workspaceId, sessionsTable); const ts = new Date().toISOString(); + // Reasoning effort + token usage aren't in the hook payload — read them from + // the rollout transcript (turn_context + token_count). `token_usage` is the + // latest turn (scoped to the current model); `token_usage_total` is the + // whole-session cumulative across every model. Codex has no assistant event, + // so this rides on every user/tool row — `turn_id` (in meta) lets a per-model + // rollup dedupe the shared per-turn snapshot. Best-effort; falls back to the + // payload model. + const modelMeta = parseCodexTurnMeta(input.transcript_path, input.model); const meta = { session_id: input.session_id, transcript_path: input.transcript_path, @@ -92,6 +101,7 @@ async function main(): Promise { model: input.model, turn_id: input.turn_id, timestamp: ts, + ...(modelMeta ?? {}), }; let entry: Record; diff --git a/src/hooks/hermes/capture.ts b/src/hooks/hermes/capture.ts index f79a5ef0..f2dd9a90 100644 --- a/src/hooks/hermes/capture.ts +++ b/src/hooks/hermes/capture.ts @@ -93,11 +93,18 @@ async function main(): Promise { const api = new DeeplakeApi(config.token, config.apiUrl, config.orgId, config.workspaceId, sessionsTable); const ts = new Date().toISOString(); + // Hermes sends `model` + `platform` nested in `extra` (everything that isn't + // tool_name/args/session_id lands there — see NousResearch/hermes-agent + // agent/shell_hooks.py). Token usage / cost are not part of the hook payload. + const model = pickString(extra.model); + const platform = pickString(extra.platform); const meta = { session_id: sessionId, cwd, hook_event_name: event, timestamp: ts, + ...(model ? { model } : {}), + ...(platform ? { usage_extra: { platform } } : {}), }; let entry: Record | null = null; diff --git a/src/hooks/shared/redact.ts b/src/hooks/shared/redact.ts index ec966204..29124614 100644 --- a/src/hooks/shared/redact.ts +++ b/src/hooks/shared/redact.ts @@ -58,6 +58,21 @@ function looksLikeSecret(tok: string): boolean { if (/^\d+$/.test(tok)) return false; // pure number if (/^[0-9a-f]+$/i.test(tok)) return false; // hex hash / git SHA / md5 if (/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(tok)) return false; // UUID + // Provider model identifiers (e.g. claude-haiku-4-5-20251001, gpt-5.6-sol, + // qwen2.5-coder-32b-instruct, us.anthropic.claude-3-5-sonnet-20241022-v2) — + // a long dated slug can otherwise trip the entropy check, and these are + // captured into every trace row's `model` field, so shredding them would + // break per-model usage rollups. Kept safe by two invariants that a random + // secret can't satisfy: lowercase only (random keys mix case) and every + // dot/dash segment capped at 12 chars (so no 24+ high-entropy run fits). + // Within those bounds we allow optional cloud/region prefixes (Bedrock etc.), + // a digit fused to the family (`qwen2`, `llama3`), and short version segments. + if ( + /^(?:(?:us|eu|apac|anthropic|amazon|bedrock|cohere|meta|mistral|google)\.)*(?:claude|gpt|o[1-9]|gemini|gemma|codex|text-embedding|dall-e|whisper|grok|deepseek|kimi|llama|mixtral|qwen|phi)[0-9]?(?:[._-][a-z0-9]{1,12})*$/.test( + tok, + ) + ) + return false; // Require a mix of character classes — random keys blend cases/digits, while // English words, dotted versions and snake_case identifiers do not. const classes = diff --git a/src/notifications/model-usage.ts b/src/notifications/model-usage.ts new file mode 100644 index 00000000..0399513a --- /dev/null +++ b/src/notifications/model-usage.ts @@ -0,0 +1,568 @@ +/** + * Model + token-usage extraction from agent transcripts. + * + * The capture hooks record one trace row per session event (user prompt, tool + * call, assistant turn). Neither Claude Code nor Codex passes model / reasoning + * effort / token counts in the hook payload, but both write them to the + * on-disk transcript the hook receives a path to: + * + * Claude Code — `~/.claude/projects//.jsonl`. Each assistant + * line is `{ type:"assistant", message:{ model, stop_reason, usage:{...} } }`. + * Reasoning effort is not in the message — it's the user's `effortLevel` + * setting, read from settings at capture time (see readClaudeEffortLevel). + * + * Codex — `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl`. A `turn_context` + * line carries `payload.model` + `payload.effort`; `event_msg` lines with + * `payload.type === "token_count"` carry `payload.info.last_token_usage` + * (this turn) and `payload.info.total_token_usage` (cumulative), each with + * `{ input_tokens, cached_input_tokens, output_tokens, + * reasoning_output_tokens, total_tokens }`. + * + * Both parsers normalize onto {@link NormalizedUsage} so a per-model rollup is a + * single `GROUP BY message->>'model'` over the sessions table. They are + * best-effort: any read/parse failure returns null and capture proceeds without + * the enrichment (never throws, never blocks a trace write). + */ + +import { closeSync, fstatSync, openSync, readFileSync, readSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import { log as _log } from "../utils/debug.js"; + +const log = (msg: string) => _log("model-usage", msg); + +/** Money cost of a turn, in the agent's billing currency (Pi / OpenClaw). */ +export interface CostBreakdown { + input?: number; + output?: number; + cache_read?: number; + cache_creation?: number; + total?: number; +} + +/** Token counts normalized across agents. Fields absent from the source are omitted. */ +export interface NormalizedUsage { + input_tokens?: number; + output_tokens?: number; + /** Tokens served from the prompt cache (Claude cache_read_input_tokens / Codex cached_input_tokens). */ + cache_read_tokens?: number; + /** Tokens written to the prompt cache (Claude cache_creation_input_tokens; Codex has none). */ + cache_creation_tokens?: number; + /** Reasoning tokens billed as output (Codex only). */ + reasoning_output_tokens?: number; + /** Grand total when the source reports one (Codex / SDK). */ + total_tokens?: number; + /** Money cost of the turn — Pi / OpenClaw report it directly (fractional). */ + cost?: CostBreakdown; +} + +/** Model / effort / token enrichment attached to a captured trace entry. */ +export interface TraceModelMeta { + model?: string; + reasoning_effort?: string | null; + /** Why the turn ended: Claude `stop_reason`, Pi/OpenClaw `stopReason`. */ + stop_reason?: string; + /** Usage for the most recent turn. */ + token_usage?: NormalizedUsage; + /** + * Cumulative session usage (Codex `total_token_usage`). This is a + * whole-session running total spanning every model used, NOT per-model — a + * session total is `MAX(total_tokens)` per session. For per-model totals, + * sum the per-turn `token_usage`, deduping by `turn_id` (each Codex turn's + * tool events share one snapshot). + */ + token_usage_total?: NormalizedUsage; + /** + * Per-harness fields that don't generalize across agents (kept faithful to + * the source rather than dropped): Claude billing detail (service_tier, + * speed, cache TTL split, server_tool_use), Codex quota (model_context_window, + * rate_limits), etc. Omitted when the agent exposes none. + */ + usage_extra?: Record; +} + +/** Token counts are non-negative integers; reject anything else before it reaches an aggregate. */ +function toNum(v: unknown): number | undefined { + return typeof v === "number" && Number.isSafeInteger(v) && v >= 0 ? v : undefined; +} + +/** Costs are non-negative finite numbers (fractional dollars); integers-only would drop them. */ +function toFloat(v: unknown): number | undefined { + return typeof v === "number" && Number.isFinite(v) && v >= 0 ? v : undefined; +} + +/** Copy only the numeric fields that are actually present, so absent keys stay absent. */ +function assign(target: NormalizedUsage, key: "input_tokens" | "output_tokens" | "cache_read_tokens" | "cache_creation_tokens" | "reasoning_output_tokens" | "total_tokens", v: unknown): void { + const n = toNum(v); + if (n !== undefined) target[key] = n; +} + +/** Normalize a Pi/OpenClaw cost object `{input, output, cacheRead, cacheWrite, total}`. */ +function normalizeCost(cost: unknown): CostBreakdown | undefined { + if (!cost || typeof cost !== "object") return undefined; + const c = cost as { input?: unknown; output?: unknown; cacheRead?: unknown; cacheWrite?: unknown; total?: unknown }; + const out: CostBreakdown = {}; + const put = (k: keyof CostBreakdown, v: unknown) => { const n = toFloat(v); if (n !== undefined) out[k] = n; }; + put("input", c.input); + put("output", c.output); + put("cache_read", c.cacheRead); + put("cache_creation", c.cacheWrite); + put("total", c.total); + return Object.keys(out).length > 0 ? out : undefined; +} + +/** + * Read only the tail of a transcript (default 1 MiB). Model / usage / stop + * reason all live in the MOST RECENT turn, which sits at the end of the file — + * so reading the whole thing on every capture event is needlessly O(filesize) + * (and O(n²) across a session for Codex, which parses on every user/tool row). + * The tail bounds each read to a constant. When the file is larger than the + * window the first (partial) line is dropped, so callers only ever see whole + * lines. Best-effort: returns null on any error. + */ +/** Read one window [size-maxBytes, size) of an open fd, returning whole lines. */ +function readWindowLines(fd: number, size: number, maxBytes: number): string[] { + if (size <= maxBytes) { + // Whole file fits — read it all; no partial-line handling needed. + const buf = Buffer.alloc(size); + const n = size > 0 ? readSync(fd, buf, 0, size, 0) : 0; + return buf.toString("utf-8", 0, n).split("\n"); + } + // Read the window PLUS one leading byte, so we can tell whether the window + // already starts at a line boundary. Buffer.alloc (zeroed) + decoding only the + // bytes actually read guards against a short read exposing stale memory. + const start = size - maxBytes - 1; + const buf = Buffer.alloc(maxBytes + 1); + const n = readSync(fd, buf, 0, maxBytes + 1, start); + const text = buf.toString("utf-8", 0, n); + // The first char is the byte BEFORE the window. If it's "\n", the window starts + // on a complete line and slicing at index 0 keeps it; otherwise the first line + // is partial and slicing drops it. Either case: cut at the first "\n". + const nl = text.indexOf("\n"); + return (nl >= 0 ? text.slice(nl + 1) : "").split("\n"); +} + +/** + * A transcript opened ONCE. `lines` is the tail (default 1 MiB window); + * `readWhole()` re-reads the entire file from the SAME fd using the size + * captured at open — so a fallback whole-file scan sees the exact same snapshot + * as the tail, closing any reopen/mutation race between the two reads. The + * caller must `close()`. + */ +interface TranscriptReader { + lines: string[]; + readWhole(): string[]; + close(): void; +} + +function openTranscript(path: string, window = 1_048_576): TranscriptReader | null { + if (!path) return null; + let fd: number; + try { + // Open directly and handle the error (incl. ENOENT) rather than an + // existsSync() pre-check — a check-then-open is a file-system race. + fd = openSync(path, "r"); + } catch (e: any) { + log(`open failed (${path}): ${e?.message ?? String(e)}`); + return null; + } + let size: number; + try { + size = fstatSync(fd).size; + } catch (e: any) { + try { closeSync(fd); } catch { /* best-effort */ } + log(`stat failed: ${e?.message ?? String(e)}`); + return null; + } + const read = (maxBytes: number): string[] => { + try { + return readWindowLines(fd, size, maxBytes); + } catch (e: any) { + log(`read failed: ${e?.message ?? String(e)}`); + return []; + } + }; + let wholeCache: string[] | undefined; + return { + lines: read(window), + readWhole: () => (wholeCache ??= read(Number.MAX_SAFE_INTEGER)), + close: () => { try { closeSync(fd); } catch { /* best-effort */ } }, + }; +} + +/** Tail lines of a transcript (default 1 MiB). Thin wrapper over {@link openTranscript}. */ +export function readTailLines(path: string, maxBytes = 1_048_576): string[] | null { + const r = openTranscript(path, maxBytes); + if (!r) return null; + try { + return r.lines; + } finally { + r.close(); + } +} + +// --------------------------------------------------------------------------- +// Pi / OpenClaw (shared SDK usage shape) +// --------------------------------------------------------------------------- + +interface SdkUsage { + input?: unknown; + output?: unknown; + cacheRead?: unknown; + cacheWrite?: unknown; + totalTokens?: unknown; + cost?: unknown; +} + +/** + * Normalize the Pi / OpenClaw SDK usage object + * (`{ input, output, cacheRead, cacheWrite, totalTokens, cost }`) onto + * {@link NormalizedUsage}, including the money `cost` sub-object. Returns + * undefined when nothing usable is present, so callers can spread it without + * emitting an empty object. + */ +export function normalizeSdkUsage(usage: unknown): NormalizedUsage | undefined { + if (!usage || typeof usage !== "object") return undefined; + const u = usage as SdkUsage; + const out: NormalizedUsage = {}; + assign(out, "input_tokens", u.input); + assign(out, "output_tokens", u.output); + assign(out, "cache_read_tokens", u.cacheRead); + assign(out, "cache_creation_tokens", u.cacheWrite); + assign(out, "total_tokens", u.totalTokens); + const cost = normalizeCost(u.cost); + if (cost) out.cost = cost; + return Object.keys(out).length > 0 ? out : undefined; +} + +/** + * Build the trace enrichment for an in-process SDK turn (Pi / OpenClaw), where + * the model, usage and stop reason arrive on the message object rather than a + * transcript file. Returns undefined when nothing usable is present. These + * runtimes expose no reasoning-effort field, so it is left unset. + */ +export function sdkTurnMeta(model: unknown, usage: unknown, stopReason?: unknown): TraceModelMeta | undefined { + const token_usage = normalizeSdkUsage(usage); + const hasModel = typeof model === "string" && model.length > 0; + const hasStop = typeof stopReason === "string" && stopReason.length > 0; + if (!hasModel && !token_usage && !hasStop) return undefined; + const meta: TraceModelMeta = {}; + if (hasModel) meta.model = model as string; + if (hasStop) meta.stop_reason = stopReason as string; + if (token_usage) meta.token_usage = token_usage; + return meta; +} + +// --------------------------------------------------------------------------- +// Claude Code +// --------------------------------------------------------------------------- + +interface ClaudeUsage { + input_tokens?: unknown; + output_tokens?: unknown; + cache_read_input_tokens?: unknown; + cache_creation_input_tokens?: unknown; + service_tier?: unknown; + speed?: unknown; + inference_geo?: unknown; + cache_creation?: { ephemeral_1h_input_tokens?: unknown; ephemeral_5m_input_tokens?: unknown }; + server_tool_use?: { web_search_requests?: unknown; web_fetch_requests?: unknown }; +} + +interface ClaudeLine { + type?: string; + cwd?: unknown; + message?: { model?: unknown; usage?: ClaudeUsage; stop_reason?: unknown }; +} + +/** + * Claude Code's reasoning effort is a user-set control (low/medium/high, with + * `ultrathink` = high), persisted as `effortLevel` in settings — NOT in the + * per-message transcript. Read it at capture time, preferring project settings + * over the user default. Best-effort: returns undefined on any miss so the + * caller falls back to null. Reflects the level configured when the turn was + * captured (an in-prompt `ultrathink` override for a single turn isn't + * recorded anywhere we can recover). + */ +const CLAUDE_EFFORT_LEVELS = new Set(["low", "medium", "high", "xhigh"]); + +export function readClaudeEffortLevel(cwd?: string): string | undefined { + const candidates: string[] = []; + if (cwd) { + candidates.push(join(cwd, ".claude", "settings.local.json")); + candidates.push(join(cwd, ".claude", "settings.json")); + } + candidates.push(join(homedir(), ".claude", "settings.json")); + for (const p of candidates) { + try { + // Read directly and let a missing file throw into the catch (no + // existsSync pre-check — that's a check-then-read race). + const j = JSON.parse(readFileSync(p, "utf-8")) as { effortLevel?: unknown }; + // Allowlist Claude's supported levels — a settings file is untrusted + // input, and an arbitrary/oversized string would poison effort analytics. + if (typeof j.effortLevel === "string") { + const level = j.effortLevel.toLowerCase(); + if (CLAUDE_EFFORT_LEVELS.has(level)) return level; + } + } catch { + // ignore unreadable / malformed settings and try the next candidate + } + } + return undefined; +} + +function normalizeClaudeUsage(u: ClaudeUsage): NormalizedUsage { + const out: NormalizedUsage = {}; + assign(out, "input_tokens", u.input_tokens); + assign(out, "output_tokens", u.output_tokens); + assign(out, "cache_read_tokens", u.cache_read_input_tokens); + assign(out, "cache_creation_tokens", u.cache_creation_input_tokens); + return out; +} + +/** + * Claude-only billing detail that doesn't generalize to other agents: the + * pricing tier / speed, the cache-creation split by TTL (1h vs 5m are priced + * differently), and billable server-side tool calls. Returns undefined when + * none are present. + */ +function claudeUsageExtra(u: ClaudeUsage): Record | undefined { + const extra: Record = {}; + if (typeof u.service_tier === "string") extra.service_tier = u.service_tier; + if (typeof u.speed === "string") extra.speed = u.speed; + if (typeof u.inference_geo === "string") extra.inference_geo = u.inference_geo; + const cc = u.cache_creation; + if (cc && typeof cc === "object") { + const ttl: Record = {}; + const a = toNum(cc.ephemeral_1h_input_tokens); if (a !== undefined) ttl.ephemeral_1h = a; + const b = toNum(cc.ephemeral_5m_input_tokens); if (b !== undefined) ttl.ephemeral_5m = b; + if (Object.keys(ttl).length) extra.cache_creation_ttl = ttl; + } + const st = u.server_tool_use; + if (st && typeof st === "object") { + const s: Record = {}; + const ws = toNum(st.web_search_requests); if (ws !== undefined) s.web_search_requests = ws; + const wf = toNum(st.web_fetch_requests); if (wf !== undefined) s.web_fetch_requests = wf; + if (Object.keys(s).length) extra.server_tool_use = s; + } + return Object.keys(extra).length > 0 ? extra : undefined; +} + +/** Reverse-scan already-read lines for the last assistant turn carrying usage. */ +function scanClaudeLines(lines: string[] | null): TraceModelMeta | null { + if (!lines) return null; + for (let i = lines.length - 1; i >= 0; i--) { + const trimmed = lines[i].trim(); + if (!trimmed) continue; + let entry: ClaudeLine; + try { + entry = JSON.parse(trimmed) as ClaudeLine; + } catch { + continue; + } + if (!entry || typeof entry !== "object") continue; // e.g. a bare `null` line + const msg = entry.message; + if (entry.type !== "assistant" || !msg || !msg.usage) continue; + const meta: TraceModelMeta = { + model: typeof msg.model === "string" ? msg.model : undefined, + // Not in the transcript message — read the configured effortLevel from + // settings (null when unset), keyed to the turn's project cwd. + reasoning_effort: readClaudeEffortLevel(typeof entry.cwd === "string" ? entry.cwd : undefined) ?? null, + token_usage: normalizeClaudeUsage(msg.usage), + }; + if (typeof msg.stop_reason === "string") meta.stop_reason = msg.stop_reason; + const extra = claudeUsageExtra(msg.usage); + if (extra) meta.usage_extra = extra; + return meta; + } + return null; +} + +/** + * Extract model + last-turn usage from a Claude Code transcript. Scans from the + * end so the returned usage belongs to the most recently completed assistant + * turn — exactly the turn whose Stop event is being captured. Reads the file + * tail for speed; falls back to the whole file if the usage-bearing line didn't + * fit the tail window (a single assistant record over the window size). Returns + * null when the file is unreadable or has no assistant line with usage. + */ +export function parseClaudeTurnMeta(transcriptPath?: string): TraceModelMeta | null { + if (!transcriptPath) return null; + const r = openTranscript(transcriptPath); + if (!r) return null; + try { + // Fast path: the last turn is in the tail. Only re-read the whole file (same + // fd/snapshot) if the usage-bearing line didn't fit the window. + return scanClaudeLines(r.lines) ?? scanClaudeLines(r.readWhole()); + } finally { + r.close(); + } +} + +// --------------------------------------------------------------------------- +// Codex +// --------------------------------------------------------------------------- + +interface CodexUsage { + input_tokens?: unknown; + cached_input_tokens?: unknown; + output_tokens?: unknown; + reasoning_output_tokens?: unknown; + total_tokens?: unknown; +} + +interface CodexRateWindow { + used_percent?: unknown; + window_minutes?: unknown; + resets_at?: unknown; +} + +interface CodexLine { + type?: string; + payload?: { + type?: string; + model?: unknown; + effort?: unknown; + info?: { + last_token_usage?: CodexUsage; + total_token_usage?: CodexUsage; + model_context_window?: unknown; + }; + rate_limits?: { primary?: CodexRateWindow; secondary?: CodexRateWindow }; + }; +} + +/** Compact Codex quota detail from a token_count event: context window + rate-limit windows. */ +function codexUsageExtra( + info: { model_context_window?: unknown } | undefined, + rateLimits: { primary?: CodexRateWindow; secondary?: CodexRateWindow } | undefined, +): Record | undefined { + const extra: Record = {}; + const win = toNum(info?.model_context_window); + if (win !== undefined) extra.model_context_window = win; + const pickWin = (w?: CodexRateWindow) => { + if (!w || typeof w !== "object") return undefined; + const o: Record = {}; + const up = toFloat(w.used_percent); if (up !== undefined) o.used_percent = up; + const wm = toNum(w.window_minutes); if (wm !== undefined) o.window_minutes = wm; + const ra = toNum(w.resets_at); if (ra !== undefined) o.resets_at = ra; + return Object.keys(o).length ? o : undefined; + }; + const primary = pickWin(rateLimits?.primary); + const secondary = pickWin(rateLimits?.secondary); + if (primary || secondary) { + const rl: Record = {}; + if (primary) rl.primary = primary; + if (secondary) rl.secondary = secondary; + extra.rate_limits = rl; + } + return Object.keys(extra).length > 0 ? extra : undefined; +} + +function normalizeCodexUsage(u: CodexUsage): NormalizedUsage { + const out: NormalizedUsage = {}; + assign(out, "input_tokens", u.input_tokens); + assign(out, "output_tokens", u.output_tokens); + assign(out, "cache_read_tokens", u.cached_input_tokens); + assign(out, "reasoning_output_tokens", u.reasoning_output_tokens); + assign(out, "total_tokens", u.total_tokens); + return out; +} + +/** + * Forward-scan already-read Codex rollout lines. `sawTurnContext` / `sawTokenCount` + * report which event types were present — the two carry different halves of the + * turn's metadata (model+effort vs usage+total+quota), so the caller only trusts + * the tail when BOTH were seen; otherwise it widens the read. + */ +function scanCodexLines( + lines: string[] | null, + fallbackModel?: string, +): { meta: TraceModelMeta | null; sawTurnContext: boolean; sawTokenCount: boolean } { + let model: string | undefined = fallbackModel; + let reasoningEffort: string | undefined; + let last: NormalizedUsage | undefined; + let total: NormalizedUsage | undefined; + let extra: Record | undefined; + let sawTurnContext = false; + let sawTokenCount = false; + + if (lines) { + for (const raw of lines) { + const trimmed = raw.trim(); + if (!trimmed) continue; + let entry: CodexLine; + try { + entry = JSON.parse(trimmed) as CodexLine; + } catch { + continue; + } + if (!entry || typeof entry !== "object") continue; // e.g. a bare `null` line + const p = entry.payload; + if (!p) continue; + if (entry.type === "turn_context") { + // A new turn context starts a new turn (possibly a new model/effort). + // Reset both to this turn's values — falling back to the hook model + // when the context omits one — so a later turn never inherits an + // earlier turn's model or effort. The previous turn's per-turn usage is + // cleared for the same reason; `total` is a session-wide cumulative and + // is intentionally preserved across turns. + sawTurnContext = true; + model = typeof p.model === "string" ? p.model : fallbackModel; + reasoningEffort = typeof p.effort === "string" ? p.effort : undefined; + last = undefined; + extra = undefined; // quota (context window / rate limits) is per-turn too + } else if (p.type === "token_count" && p.info) { + sawTokenCount = true; + if (p.info.last_token_usage) last = normalizeCodexUsage(p.info.last_token_usage); + if (p.info.total_token_usage) total = normalizeCodexUsage(p.info.total_token_usage); + extra = codexUsageExtra(p.info, p.rate_limits); // latest token_count wins (may clear) + } + } + } + + const empty = model === undefined && reasoningEffort === undefined && !last && !total && !extra; + if (empty) return { meta: null, sawTurnContext, sawTokenCount }; + const meta: TraceModelMeta = {}; + if (model !== undefined) meta.model = model; + if (reasoningEffort !== undefined) meta.reasoning_effort = reasoningEffort; + if (last) meta.token_usage = last; + if (total) meta.token_usage_total = total; + if (extra) meta.usage_extra = extra; + return { meta, sawTurnContext, sawTokenCount }; +} + +/** + * Extract model + reasoning effort + latest token usage from a Codex rollout. + * Keeps the last `turn_context` (model/effort) and last `token_count` (per-turn + * + cumulative usage + quota). Reads the file tail for speed; the two event + * types carry different halves of the metadata, so if the tail is missing + * EITHER (a turn with >window bytes between them), it re-scans the whole file — + * whose result is authoritative. `fallbackModel` is the hook payload's `model`. + */ +export function parseCodexTurnMeta( + transcriptPath?: string | null, + fallbackModel?: string, +): TraceModelMeta | null { + if (!transcriptPath) return scanCodexLines(null, fallbackModel).meta; + const r = openTranscript(transcriptPath); + if (!r) return scanCodexLines(null, fallbackModel).meta; + try { + const tail = scanCodexLines(r.lines, fallbackModel); + // Trust the tail only if it holds the COMPLETE latest state: model/effort + // (turn_context), per-turn usage + quota (token_count), AND the cumulative + // total — which the whole-file scan carries across turns and an in-tail + // token_count might not include if an earlier one set it. + if (tail.sawTurnContext && tail.sawTokenCount && tail.meta?.token_usage_total !== undefined) { + return tail.meta; + } + // Widen to the whole file — SAME fd + size captured at open, so tail and + // whole are one snapshot (no reopen/mutation race). Trust the whole scan + // only if it saw Codex events; otherwise keep what the tail extracted. + const whole = scanCodexLines(r.readWhole(), fallbackModel); + if (!whole.sawTurnContext && !whole.sawTokenCount) return tail.meta; + return whole.meta ?? tail.meta; + } finally { + r.close(); + } +} diff --git a/tests/claude-code/skillify-session-start-injection.test.ts b/tests/claude-code/skillify-session-start-injection.test.ts index df28df2c..e8d98564 100644 --- a/tests/claude-code/skillify-session-start-injection.test.ts +++ b/tests/claude-code/skillify-session-start-injection.test.ts @@ -289,12 +289,17 @@ describe("OpenClaw skillify worker (mining) wiring", () => { expect(src).toMatch(/const skillifySpawnedFor = new Set\(\)/); expect(src).toMatch(/if \(!skillifySpawnedFor\.has\(sid\)\)/); expect(src).toMatch(/skillifySpawnedFor\.add\(sid\)/); - // agent_end hook calls it after the capture loop. Distance bumped - // from 500→1500 to accommodate the per-runtime spawn-dedup comment - // block landed for #100 between `Auto-captured` and the spawn site. - // 3500→4000 for the embed-on-capture wiring landed for #178 between - // `agent_end` and `Auto-captured` (tryEmbedStandalone + comment block). - expect(src).toMatch(/agent_end[\s\S]{0,4000}Auto-captured[\s\S]{0,1500}spawnOpenclawSkillifyWorker/); + // The skillify worker is spawned from the agent_end capture hook, after + // the "Auto-captured" log. Anchor on the real constructs and assert their + // source order — robust to unrelated code landing between them, unlike a + // magic character-count gap (CodeRabbit on #320). + const iAgentEnd = src.search(/hook\(\s*["']agent_end["']/); + const iAutoCaptured = src.indexOf("Auto-captured", iAgentEnd); + // The CALL site, after the log — not the earlier function definition. + const iSpawnCall = src.indexOf("spawnOpenclawSkillifyWorker(", iAutoCaptured); + expect(iAgentEnd).toBeGreaterThanOrEqual(0); + expect(iAutoCaptured).toBeGreaterThan(iAgentEnd); + expect(iSpawnCall).toBeGreaterThan(iAutoCaptured); // install: "global" — no per-project cwd, skills land under ~/.claude/skills/ expect(src).toMatch(/install:\s*"global"/); }); diff --git a/tests/hermes/hermes-capture-hook.test.ts b/tests/hermes/hermes-capture-hook.test.ts index 1611a4a9..d5e322d0 100644 --- a/tests/hermes/hermes-capture-hook.test.ts +++ b/tests/hermes/hermes-capture-hook.test.ts @@ -238,6 +238,24 @@ describe("hermes capture hook — post_llm_call (assistant message)", () => { expect(queryMock).not.toHaveBeenCalled(); expect(debugLogMock).toHaveBeenCalledWith(expect.stringContaining("no response found")); }); + + it("captures model + platform from extra (Hermes nests them there)", async () => { + stdinMock.mockResolvedValue({ + hook_event_name: "post_llm_call", + session_id: "sid", + extra: { response: "done", model: "hermes-4-405b", platform: "nousresearch" }, + }); + await runHook(); + const sql = queryMock.mock.calls[0][0] as string; + expect(sql).toContain('"model":"hermes-4-405b"'); + expect(sql).toContain('"usage_extra":{"platform":"nousresearch"}'); + }); + + it("omits model when extra carries none (no fabrication)", async () => { + stdinMock.mockResolvedValue({ hook_event_name: "post_llm_call", session_id: "sid", extra: { response: "done" } }); + await runHook(); + expect((queryMock.mock.calls[0][0] as string)).not.toContain('"model"'); + }); }); describe("hermes capture hook — unknown / failure paths", () => { diff --git a/tests/shared/notifications-model-usage.test.ts b/tests/shared/notifications-model-usage.test.ts new file mode 100644 index 00000000..28b83901 --- /dev/null +++ b/tests/shared/notifications-model-usage.test.ts @@ -0,0 +1,543 @@ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { mkdirSync } from "node:fs"; +import { + parseClaudeTurnMeta, + parseCodexTurnMeta, + normalizeSdkUsage, + sdkTurnMeta, + readClaudeEffortLevel, + readTailLines, +} from "../../src/notifications/model-usage.js"; + +let TEMP_DIR = ""; + +beforeEach(() => { + TEMP_DIR = mkdtempSync(join(tmpdir(), "hivemind-model-usage-test-")); +}); + +afterEach(() => { + rmSync(TEMP_DIR, { recursive: true, force: true }); +}); + +function writeTranscript(lines: object[]): string { + const file = join(TEMP_DIR, "transcript.jsonl"); + writeFileSync(file, lines.map((l) => JSON.stringify(l)).join("\n") + "\n", "utf-8"); + return file; +} + +// Real Claude Code assistant line shape (see transcript sample in model-usage.ts docstring). +function claudeAssistantLine(model: string, usage: Record) { + return { + type: "assistant", + message: { model, role: "assistant", content: [], usage }, + }; +} + +// Real Codex rollout line shapes. +function codexTurnContext(model: string, effort: string) { + return { type: "turn_context", payload: { model, effort } }; +} +function codexTokenCount(last: Record, total: Record) { + return { + type: "event_msg", + payload: { type: "token_count", info: { last_token_usage: last, total_token_usage: total } }, + }; +} + +describe("parseClaudeTurnMeta", () => { + it("returns null for missing path / no path", () => { + expect(parseClaudeTurnMeta(undefined)).toBeNull(); + expect(parseClaudeTurnMeta("/tmp/does-not-exist-hivemind.jsonl")).toBeNull(); + }); + + it("extracts model + normalized usage from the last assistant turn", () => { + const file = writeTranscript([ + claudeAssistantLine("claude-sonnet-4-6", { + input_tokens: 100, + output_tokens: 10, + cache_read_input_tokens: 5, + cache_creation_input_tokens: 7, + }), + { type: "user", message: { role: "user", content: [] } }, + claudeAssistantLine("claude-opus-4-8", { + input_tokens: 19088, + output_tokens: 404, + cache_read_input_tokens: 15547, + cache_creation_input_tokens: 18853, + }), + ]); + const meta = parseClaudeTurnMeta(file); + // Last assistant turn wins (reverse scan), not the first. + expect(meta?.model).toBe("claude-opus-4-8"); + // reasoning_effort is read from settings (null when unset); this token-focused + // test stays hermetic by only asserting the shape, not an ambient value. + expect(meta?.reasoning_effort === null || typeof meta?.reasoning_effort === "string").toBe(true); + expect(meta?.token_usage).toEqual({ + input_tokens: 19088, + output_tokens: 404, + cache_read_tokens: 15547, + cache_creation_tokens: 18853, + }); + }); + + it("omits absent usage fields rather than defaulting them to 0", () => { + const file = writeTranscript([ + claudeAssistantLine("claude-opus-4-8", { input_tokens: 50, output_tokens: 3 }), + ]); + const meta = parseClaudeTurnMeta(file); + expect(meta?.token_usage).toEqual({ input_tokens: 50, output_tokens: 3 }); + expect(meta?.token_usage).not.toHaveProperty("cache_read_tokens"); + }); + + it("skips malformed lines and assistant lines without usage", () => { + // First line is not JSON; second is an assistant line missing usage; third + // is the valid one the reverse scan should return. + const file = join(TEMP_DIR, "transcript.jsonl"); + const lines = [ + "not json", + "null", // valid JSON but not an object — must not throw + JSON.stringify({ type: "assistant", message: { model: "claude-opus-4-8", role: "assistant" } }), + JSON.stringify(claudeAssistantLine("claude-opus-4-8", { input_tokens: 1, output_tokens: 1 })), + ]; + writeFileSync(file, lines.join("\n") + "\n", "utf-8"); + const meta = parseClaudeTurnMeta(file); + expect(meta?.token_usage).toEqual({ input_tokens: 1, output_tokens: 1 }); + }); +}); + +describe("parseCodexTurnMeta", () => { + it("falls back to payload model when the transcript is missing", () => { + const meta = parseCodexTurnMeta(undefined, "gpt-5.6-sol"); + expect(meta?.model).toBe("gpt-5.6-sol"); + expect(meta?.token_usage).toBeUndefined(); + }); + + it("returns null when nothing is available", () => { + expect(parseCodexTurnMeta(undefined, undefined)).toBeNull(); + }); + + it("extracts model, reasoning effort, last + total usage from the rollout", () => { + const file = writeTranscript([ + codexTurnContext("gpt-5.5", "medium"), + codexTokenCount( + { input_tokens: 20131, cached_input_tokens: 9984, output_tokens: 154, reasoning_output_tokens: 0, total_tokens: 20285 }, + { input_tokens: 20131, cached_input_tokens: 9984, output_tokens: 154, reasoning_output_tokens: 0, total_tokens: 20285 }, + ), + codexTokenCount( + { input_tokens: 32758, cached_input_tokens: 32128, output_tokens: 79, reasoning_output_tokens: 0, total_tokens: 32837 }, + { input_tokens: 212981, cached_input_tokens: 196736, output_tokens: 3427, reasoning_output_tokens: 160, total_tokens: 216408 }, + ), + ]); + const meta = parseCodexTurnMeta(file, "payload-model"); + // turn_context model overrides the payload fallback. + expect(meta?.model).toBe("gpt-5.5"); + expect(meta?.reasoning_effort).toBe("medium"); + // Latest token_count wins for both per-turn and cumulative. + expect(meta?.token_usage).toEqual({ + input_tokens: 32758, + output_tokens: 79, + cache_read_tokens: 32128, + reasoning_output_tokens: 0, + total_tokens: 32837, + }); + expect(meta?.token_usage_total).toEqual({ + input_tokens: 212981, + output_tokens: 3427, + cache_read_tokens: 196736, + reasoning_output_tokens: 160, + total_tokens: 216408, + }); + }); + + it("does not attach a prior turn's usage to a new turn_context (model change)", () => { + // Turn A completes with a token_count, then turn B opens with a new model + // BEFORE its own token_count arrives (the UserPromptSubmit at turn start). + const file = writeTranscript([ + codexTurnContext("gpt-5.5", "medium"), + codexTokenCount( + { input_tokens: 100, output_tokens: 10, total_tokens: 110 }, + { input_tokens: 100, output_tokens: 10, total_tokens: 110 }, + ), + codexTurnContext("gpt-5.6-sol", "high"), + ]); + const meta = parseCodexTurnMeta(file, "fb"); + expect(meta?.model).toBe("gpt-5.6-sol"); + expect(meta?.reasoning_effort).toBe("high"); + // Turn A's tokens must NOT be attributed to turn B's model. + expect(meta?.token_usage).toBeUndefined(); + // Session cumulative is preserved across the turn boundary. + expect(meta?.token_usage_total).toEqual({ input_tokens: 100, output_tokens: 10, total_tokens: 110 }); + }); + + it("resets model + effort per turn, falling back to the hook model when a context omits them", () => { + const file = writeTranscript([ + codexTurnContext("gpt-5.5", "medium"), + codexTokenCount({ input_tokens: 9, output_tokens: 1, total_tokens: 10 }, { input_tokens: 9, output_tokens: 1, total_tokens: 10 }), + { type: "turn_context", payload: {} }, // new turn with no model/effort + ]); + const meta = parseCodexTurnMeta(file, "hook-model"); + expect(meta?.model).toBe("hook-model"); // not the stale gpt-5.5 + expect(meta?.reasoning_effort).toBeUndefined(); // not the stale "medium" + }); + + it("rejects negative / fractional token counts", () => { + const file = writeTranscript([ + codexTurnContext("gpt-5.5", "low"), + codexTokenCount( + { input_tokens: -5, output_tokens: 3.5, total_tokens: 10 }, + { input_tokens: 10, output_tokens: 2, total_tokens: 12 }, + ), + ]); + const meta = parseCodexTurnMeta(file, "fb"); + // input_tokens (negative) and output_tokens (fractional) dropped; total kept. + expect(meta?.token_usage).toEqual({ total_tokens: 10 }); + }); + + it("survives a bare null JSONL line (best-effort contract)", () => { + const file = join(TEMP_DIR, "transcript.jsonl"); + const lines = [ + "null", + JSON.stringify(codexTurnContext("gpt-5.5", "low")), + JSON.stringify(codexTokenCount({ input_tokens: 3, output_tokens: 1, total_tokens: 4 }, { input_tokens: 3, output_tokens: 1, total_tokens: 4 })), + ]; + writeFileSync(file, lines.join("\n") + "\n", "utf-8"); + const meta = parseCodexTurnMeta(file, "fb"); + expect(meta?.model).toBe("gpt-5.5"); + expect(meta?.token_usage?.total_tokens).toBe(4); + }); + + it("keeps payload model when the rollout has token counts but no turn_context", () => { + const file = writeTranscript([ + codexTokenCount( + { input_tokens: 5, output_tokens: 2, total_tokens: 7 }, + { input_tokens: 5, output_tokens: 2, total_tokens: 7 }, + ), + ]); + const meta = parseCodexTurnMeta(file, "gpt-5.6-sol"); + expect(meta?.model).toBe("gpt-5.6-sol"); + expect(meta?.reasoning_effort).toBeUndefined(); + expect(meta?.token_usage?.total_tokens).toBe(7); + }); +}); + +describe("normalizeSdkUsage / sdkTurnMeta (Pi + OpenClaw)", () => { + it("normalizes the real Pi usage shape incl. fractional cost", () => { + // From a real ~/.pi transcript: {input, output, cacheRead, cacheWrite, totalTokens, cost}. + const u = { input: 12456, output: 15, cacheRead: 0, cacheWrite: 0, totalTokens: 12471, cost: { input: 0.06228, output: 0.00045, cacheRead: 0, cacheWrite: 0, total: 0.06273 } }; + expect(normalizeSdkUsage(u)).toEqual({ + input_tokens: 12456, + output_tokens: 15, + cache_read_tokens: 0, + cache_creation_tokens: 0, + total_tokens: 12471, + cost: { input: 0.06228, output: 0.00045, cache_read: 0, cache_creation: 0, total: 0.06273 }, + }); + }); + + it("normalizes the real OpenClaw usage shape and maps cacheWrite -> cache_creation", () => { + const u = { input: 28621, output: 806, cacheRead: 4, cacheWrite: 9, totalTokens: 29427 }; + expect(normalizeSdkUsage(u)).toEqual({ + input_tokens: 28621, + output_tokens: 806, + cache_read_tokens: 4, + cache_creation_tokens: 9, + total_tokens: 29427, + }); + }); + + it("returns undefined for empty / invalid usage", () => { + expect(normalizeSdkUsage(undefined)).toBeUndefined(); + expect(normalizeSdkUsage({})).toBeUndefined(); + // Negative + fractional both rejected -> no valid keys -> undefined. + expect(normalizeSdkUsage({ input: -1, output: 2.5 })).toBeUndefined(); + }); + + it("builds sdkTurnMeta with model + token_usage, no reasoning effort", () => { + const meta = sdkTurnMeta("gpt-5.5", { input: 10, output: 3, totalTokens: 13 }); + expect(meta).toEqual({ model: "gpt-5.5", token_usage: { input_tokens: 10, output_tokens: 3, total_tokens: 13 } }); + expect(meta).not.toHaveProperty("reasoning_effort"); + }); + + it("returns undefined when neither model nor usage is present", () => { + expect(sdkTurnMeta(undefined, undefined)).toBeUndefined(); + expect(sdkTurnMeta("", {})).toBeUndefined(); + }); + + it("keeps model even when usage is absent", () => { + expect(sdkTurnMeta("claude-via-openclaw", null)).toEqual({ model: "claude-via-openclaw" }); + }); + + it("carries stop_reason and rejects a fractional token count but keeps fractional cost", () => { + const meta = sdkTurnMeta("gpt-5.5", { input: 10, output: 3, totalTokens: 13, cost: { total: 0.0627 } }, "stop"); + expect(meta).toEqual({ + model: "gpt-5.5", + stop_reason: "stop", + token_usage: { input_tokens: 10, output_tokens: 3, total_tokens: 13, cost: { total: 0.0627 } }, + }); + }); + + it("emits stop_reason alone when only it is present", () => { + expect(sdkTurnMeta(undefined, undefined, "toolUse")).toEqual({ stop_reason: "toolUse" }); + }); +}); + +describe("parseClaudeTurnMeta — stop_reason + usage_extra", () => { + const tmp = () => join(TEMP_DIR, "transcript.jsonl"); + it("extracts stop_reason and Claude billing extras (tier/speed/cache-ttl/server-tools)", () => { + const line = { + type: "assistant", + message: { + model: "claude-opus-4-8", + stop_reason: "end_turn", + usage: { + input_tokens: 100, output_tokens: 20, + cache_read_input_tokens: 5, cache_creation_input_tokens: 7, + service_tier: "standard", speed: "standard", inference_geo: "not_available", + cache_creation: { ephemeral_1h_input_tokens: 7, ephemeral_5m_input_tokens: 0 }, + server_tool_use: { web_search_requests: 2, web_fetch_requests: 1 }, + }, + }, + }; + writeFileSync(tmp(), JSON.stringify(line) + "\n", "utf-8"); + const meta = parseClaudeTurnMeta(tmp()); + expect(meta?.stop_reason).toBe("end_turn"); + expect(meta?.token_usage).toEqual({ input_tokens: 100, output_tokens: 20, cache_read_tokens: 5, cache_creation_tokens: 7 }); + expect(meta?.usage_extra).toEqual({ + service_tier: "standard", + speed: "standard", + inference_geo: "not_available", + cache_creation_ttl: { ephemeral_1h: 7, ephemeral_5m: 0 }, + server_tool_use: { web_search_requests: 2, web_fetch_requests: 1 }, + }); + }); + + it("omits usage_extra when the assistant turn has no extra billing fields", () => { + const line = { type: "assistant", message: { model: "claude-opus-4-8", usage: { input_tokens: 1, output_tokens: 1 } } }; + writeFileSync(tmp(), JSON.stringify(line) + "\n", "utf-8"); + const meta = parseClaudeTurnMeta(tmp()); + expect(meta).not.toHaveProperty("usage_extra"); + expect(meta).not.toHaveProperty("stop_reason"); + }); +}); + +describe("readTailLines", () => { + const f = () => join(TEMP_DIR, "tail.jsonl"); + it("returns all lines when the file fits in the window", () => { + writeFileSync(f(), "a\nb\nc\n", "utf-8"); + expect(readTailLines(f(), 1024)).toEqual(["a", "b", "c", ""]); + }); + + it("reads only the tail and drops the partial first line when over the window", () => { + // 5 lines; a small window that starts mid-file. The first surviving line + // must be a WHOLE line (partial leading line dropped). + writeFileSync(f(), "L1-oldest\nL2\nL3\nL4\nL5-newest\n", "utf-8"); + const lines = readTailLines(f(), 12); // ~last 12 bytes -> mid "L4"/"L5" + expect(lines).not.toBeNull(); + // No partial line: every returned non-empty line is one of the originals. + for (const ln of lines!.filter(Boolean)) { + expect(["L1-oldest", "L2", "L3", "L4", "L5-newest"]).toContain(ln); + } + // The newest line is always present (it's at the very end). + expect(lines).toContain("L5-newest"); + // The oldest is NOT in a 12-byte tail. + expect(lines).not.toContain("L1-oldest"); + }); + + it("returns null for a missing file", () => { + expect(readTailLines(join(TEMP_DIR, "nope.jsonl"))).toBeNull(); + }); + + it("keeps a COMPLETE line when the window starts exactly at a line boundary", () => { + // "AAAA\nBBBB\n" is 10 bytes; a 5-byte window starts exactly at "BBBB". + // The boundary byte before the window is "\n", so BBBB is whole and kept. + writeFileSync(f(), "AAAA\nBBBB\n", "utf-8"); + expect(readTailLines(f(), 5)).toEqual(["BBBB", ""]); + }); + + it("still extracts from a real-shaped transcript whose relevant line is in the tail", () => { + // Big filler older turn, then the assistant turn with usage at the very end. + const filler = Array.from({ length: 50 }, (_, i) => JSON.stringify({ type: "user", n: i, pad: "x".repeat(200) })).join("\n"); + const last = JSON.stringify(claudeAssistantLine("claude-opus-4-8", { input_tokens: 7, output_tokens: 8 })); + writeFileSync(f(), filler + "\n" + last + "\n", "utf-8"); + const meta = parseClaudeTurnMeta(f()); // default 1 MiB window covers it + expect(meta?.token_usage).toEqual({ input_tokens: 7, output_tokens: 8 }); + }); + + it("falls back to the whole file when the assistant record exceeds the tail window", () => { + // A single assistant record larger than the 1 MiB tail: its start (with the + // usage) is outside the window, so the tail scan finds a partial/nothing and + // parseClaudeTurnMeta must re-read the whole file to recover the usage. + const huge = { type: "assistant", message: { model: "claude-opus-4-8", usage: { input_tokens: 11, output_tokens: 22 }, content: "y".repeat(1_200_000) } }; + writeFileSync(f(), JSON.stringify(huge) + "\n", "utf-8"); + const meta = parseClaudeTurnMeta(f()); + expect(meta?.model).toBe("claude-opus-4-8"); + expect(meta?.token_usage).toEqual({ input_tokens: 11, output_tokens: 22 }); + }); +}); + +describe("readClaudeEffortLevel", () => { + it("reads effortLevel from a project settings file, project overriding", () => { + const proj = join(TEMP_DIR, "proj"); + mkdirSync(join(proj, ".claude"), { recursive: true }); + writeFileSync(join(proj, ".claude", "settings.json"), JSON.stringify({ effortLevel: "high" }), "utf-8"); + expect(readClaudeEffortLevel(proj)).toBe("high"); + // settings.local.json wins over settings.json + writeFileSync(join(proj, ".claude", "settings.local.json"), JSON.stringify({ effortLevel: "low" }), "utf-8"); + expect(readClaudeEffortLevel(proj)).toBe("low"); + }); + + it("returns undefined when no settings file has effortLevel", () => { + const empty = join(TEMP_DIR, "empty"); + mkdirSync(empty, { recursive: true }); + // A cwd with no .claude dir — only the (possibly absent) user settings apply. + const r = readClaudeEffortLevel(empty); + expect(r === undefined || typeof r === "string").toBe(true); // never throws + }); + + it("ignores malformed settings JSON and falls through", () => { + const bad = join(TEMP_DIR, "bad"); + mkdirSync(join(bad, ".claude"), { recursive: true }); + writeFileSync(join(bad, ".claude", "settings.json"), "{ not valid json", "utf-8"); + expect(() => readClaudeEffortLevel(bad)).not.toThrow(); + }); + + it("rejects an effortLevel outside Claude's supported levels", () => { + const evil = join(TEMP_DIR, "evil"); + mkdirSync(join(evil, ".claude"), { recursive: true }); + // Only this settings file on the path (project), and its value is bogus → + // must NOT be returned (falls through; user settings may still apply, but + // the bogus project value itself is never accepted). + writeFileSync(join(evil, ".claude", "settings.json"), JSON.stringify({ effortLevel: "A".repeat(5000) }), "utf-8"); + const r = readClaudeEffortLevel(evil); + expect(r).not.toBe("A".repeat(5000)); + expect(r === undefined || CLAUDE_LEVELS.has(r)).toBe(true); + }); + + it("normalizes case to a supported level", () => { + const up = join(TEMP_DIR, "up"); + mkdirSync(join(up, ".claude"), { recursive: true }); + writeFileSync(join(up, ".claude", "settings.local.json"), JSON.stringify({ effortLevel: "HIGH" }), "utf-8"); + expect(readClaudeEffortLevel(up)).toBe("high"); + }); +}); + +const CLAUDE_LEVELS = new Set(["low", "medium", "high", "xhigh"]); + +describe("parseCodexTurnMeta — usage_extra quota", () => { + it("extracts model_context_window and rate_limits from the latest token_count", () => { + const file = writeTranscript([ + codexTurnContext("gpt-5.5", "medium"), + { + type: "event_msg", + payload: { + type: "token_count", + info: { + last_token_usage: { input_tokens: 10, output_tokens: 2, total_tokens: 12 }, + total_token_usage: { input_tokens: 10, output_tokens: 2, total_tokens: 12 }, + model_context_window: 258400, + }, + rate_limits: { + primary: { used_percent: 11.0, window_minutes: 10080, resets_at: 1784782051 }, + secondary: { used_percent: 2.0, window_minutes: 300, resets_at: 1779231430 }, + }, + }, + }, + ]); + const meta = parseCodexTurnMeta(file, "fb"); + expect(meta?.usage_extra).toEqual({ + model_context_window: 258400, + rate_limits: { + primary: { used_percent: 11.0, window_minutes: 10080, resets_at: 1784782051 }, + secondary: { used_percent: 2.0, window_minutes: 300, resets_at: 1779231430 }, + }, + }); + }); + + it("drops invalid quota values (negative / fractional) but keeps valid ones", () => { + const file = writeTranscript([ + codexTurnContext("gpt-5.5", "medium"), + { + type: "event_msg", + payload: { + type: "token_count", + info: { + last_token_usage: { input_tokens: 1, output_tokens: 1, total_tokens: 2 }, + model_context_window: 3.5, // fractional -> dropped + }, + rate_limits: { + primary: { used_percent: -5, window_minutes: 1.5, resets_at: 100 }, // only resets_at valid + secondary: { used_percent: 2, window_minutes: 300, resets_at: 1 }, // all valid + }, + }, + }, + ]); + const meta = parseCodexTurnMeta(file, "fb"); + expect(meta?.usage_extra).toEqual({ + rate_limits: { + primary: { resets_at: 100 }, + secondary: { used_percent: 2, window_minutes: 300, resets_at: 1 }, + }, + }); + expect(meta?.usage_extra).not.toHaveProperty("model_context_window"); + }); + + it("falls back to the whole file when turn_context is pushed out of the tail by a huge record", () => { + // turn_context (model/effort) ... >1 MiB event ... token_count (usage) at end. + // The 1 MiB tail catches only the token_count, so parseCodexTurnMeta must + // re-read the whole file to recover the model/effort from turn_context. + const file = writeTranscript([ + codexTurnContext("gpt-5.5", "high"), + { type: "event_msg", payload: { type: "note", pad: "z".repeat(1_200_000) } }, + codexTokenCount( + { input_tokens: 9, output_tokens: 1, total_tokens: 10 }, + { input_tokens: 9, output_tokens: 1, total_tokens: 10 }, + ), + ]); + const meta = parseCodexTurnMeta(file, "payload-fallback"); + expect(meta?.model).toBe("gpt-5.5"); // recovered from turn_context, not the payload fallback + expect(meta?.reasoning_effort).toBe("high"); + expect(meta?.token_usage?.total_tokens).toBe(10); + }); + + it("recovers a cumulative total set before the tail window via the whole-file scan", () => { + // First token_count carries total; a >1 MiB record follows; then a new turn + // whose token_count OMITS total. The tail sees both signals but no total, so + // parseCodexTurnMeta must widen to recover token_usage_total from the file. + const file = writeTranscript([ + codexTurnContext("gpt-5.5", "medium"), + codexTokenCount( + { input_tokens: 5, output_tokens: 5, total_tokens: 10 }, + { input_tokens: 5, output_tokens: 5, total_tokens: 500 }, + ), + { type: "event_msg", payload: { type: "note", pad: "q".repeat(1_200_000) } }, + codexTurnContext("gpt-5.5", "medium"), + // token_count WITHOUT total_token_usage + { type: "event_msg", payload: { type: "token_count", info: { last_token_usage: { input_tokens: 1, output_tokens: 1, total_tokens: 2 } } } }, + ]); + const meta = parseCodexTurnMeta(file, "fb"); + expect(meta?.token_usage_total?.total_tokens).toBe(500); // preserved from the first token_count + }); + + it("does not carry a prior turn's quota into a new turn_context (model change)", () => { + const file = writeTranscript([ + codexTurnContext("gpt-5.5", "medium"), + { + type: "event_msg", + payload: { + type: "token_count", + info: { last_token_usage: { input_tokens: 1, output_tokens: 1, total_tokens: 2 }, model_context_window: 258400 }, + rate_limits: { primary: { used_percent: 11, window_minutes: 10080, resets_at: 1 } }, + }, + }, + codexTurnContext("gpt-5.6-sol", "high"), // new turn, no token_count after it + ]); + const meta = parseCodexTurnMeta(file, "fb"); + expect(meta?.model).toBe("gpt-5.6-sol"); + // Quota from gpt-5.5's turn must NOT attach to gpt-5.6-sol. + expect(meta).not.toHaveProperty("usage_extra"); + }); +}); diff --git a/tests/shared/redact.test.ts b/tests/shared/redact.test.ts index f3220142..324c243f 100644 --- a/tests/shared/redact.test.ts +++ b/tests/shared/redact.test.ts @@ -220,6 +220,35 @@ describe("redactSecrets — high-entropy backstop", () => { it("does NOT mask a long decimal number", () => { expect(redactSecrets("count=123456789012345678901234567890")).toContain("123456789012345678901234567890"); }); + + it("does NOT mask provider model identifiers (stored in trace rows)", () => { + // These are captured into every trace row's `model` field; masking a long + // dated slug would break per-model token rollups. Bare token in / bare + // token out — exact, no redaction. + for (const m of [ + "claude-haiku-4-5-20251001", + "claude-3-5-sonnet-20241022", + "gemini-3-flash-preview", + "qwen2.5-coder-32b-instruct", + "llama3.1-405b-instruct", + "us.anthropic.claude-3-5-sonnet-20241022-v2", + ]) { + expect(redactSecrets(m)).toBe(m); + } + }); + + it("STILL masks a high-entropy secret that wears a model-name prefix", () => { + // The model-id exemption must not become a bypass: a random key with a + // `gpt-`/`claude-` prefix has an over-long or mixed-case segment and must + // still be redacted to exactly the mask. + for (const s of [ + "gpt-AbCdEfGhIjKlMnOpQrStUvWxYz123456", + "claude-Zk9QW3mLpX7vNbR2tYcH8dFgJsK4uE1a", + "gpt-abcdefghijklmnopqrstuvwxyz0123456789", + ]) { + expect(redactSecrets(s)).toBe(MASK); + } + }); }); describe("redactSecrets — idempotency & multi-secret", () => {