A runnable workbench for Claude-style agent harnesses. It helps you decide whether a task deserves an agent, render a structured system prompt, lint tool contracts, run local transcript evals, import external agent traces, and ask Claude to judge trace quality when a live key is available.
Use it when an agent is failing for boring reasons: fuzzy tool names, broad schemas, weak stop criteria, missing held-out cases, unclear reasoning summaries, or no baseline. The repo turns those failure modes into repeatable checks.
The value bar is adversarially-confirmed to add value. A prompt, tool, harness, or matrix result is promoted only when it names the value claim, compares against a baseline, clears the threshold, and survives an adversarial check with no open objections.
The proof loop is local first: define a risky tool or harness boundary, run the same cases through baseline and candidate surfaces, inspect trace behavior, keep the receipts, and promote only the smallest measured improvement that survives held-out checks.
Note
This page is optimized for founders and maintainers first. The exhaustive harness gates, retained surfaces, and LLM-readable audit details are preserved at the bottom in a collapsible section.
Tip
Watch for the Supabase coverage audit, the DDL and RLS optimization cells, retained live result summary, packet checks, and final share URL. For playback controls, open demo.mp4.
The featured example is the Supabase MCP Database Tool Tuning full PR/evidence bundle. It shows an agent boundary that is easy to understand and high impact: schema-changing SQL should use apply_migration, not raw execute_sql. The demo is generated by scripts/render_demo_gif.py from demo.tape.
| Audience | Send this |
|---|---|
| Supabase maintainers | Supabase finding, retained PR/evidence bundle, and tool tuning note |
| Any MCP founder | Founder Findings |
| A reviewer who wants the proof trail | Confirmed Improvements |
| Someone evaluating the workflow | Demo GIF or demo.mp4 |
| Current frontier receipts | Frontier stress receipts across OpenAI, Gemini, and retained Anthropic Opus attempts |
python3.11 -m venv .venv
source .venv/bin/activate
pip install -e .
python -m claude_agent_harness_opt render recipes/agentic_search.json
python -m claude_agent_harness_opt score recipes/agentic_search.json
python -m claude_agent_harness_opt lint-tools recipes/agentic_search.json
python -m claude_agent_harness_opt eval evals/examples/search_answer.json
python -m claude_agent_harness_opt review-trace evals/examples/agent_trace_good.jsonThose commands are keyless except for live Claude judging and cross-provider sweeps. For the wider surface, start with Setup and Techniques.
The kit covers four jobs:
- Prompt and tool design: task-fit scoring, prompt rendering, tool-description linting, tool budget checks, stop criteria, fallback behavior, and destructive-action guardrails.
- Trace and eval review: answer evals, tool-use evals, final-state evals, trace suites, audit bundles, and optional Claude judging over visible reasoning summaries and tool outputs.
- Harness import and comparison: adapters for Claude Messages, Agent SDK exports, IDE-agent events, live harness CLI runs, model matrices, and surface snapshots.
- Tool-contract optimization: baseline matrices, candidate tool-description variants, held-out checks, reports, and upstream PR packets that promote only measured improvements.
- Capture a real tool or skill surface as a matrix.
- Run baseline and tuned variants across the same cases.
- Promote only changes that clear the adversarially-confirmed value bar.
- Package the finding as a public, clickable folder with result receipts and reproduction commands.
Hill Descending And Climbing underpins the repo. Hill descending finds the weak boundary by inventorying tools, skills, schemas, resources, traces, and support signals, then turning them into adversarial matrix cases. Hill climbing starts only after repeatable baseline failures, and promotes the smallest measured change that clears held-out checks.
| Need | Open |
|---|---|
| Send a founder one clean finding | Founder Findings |
| See every promoted win and guardrail | Confirmed Improvements |
| Review the YC P2026 sweep | YC P2026 MCP Sweep |
| Learn the tool-writing standard | Tool Writing Best Practices |
| Decide whether a workflow is a skill or a tool | Skills vs Tools |
| Run the broader public MCP sweep | Public MCP Sweep |
| Audit retained surfaces and gates | Surface Inventory |
| Review current frontier sweeps | Available-frontier stress receipts |
Every link in this table is public and can be sent directly. The finding folder is the founder-facing summary. The full bundle points at the complete retained artifact set for that target, including a PR/evidence packet where one has been generated.
| Target | Before | After | Result |
|---|---|---|---|
| gstack | gstack_stock_skill_descriptions scored 0.975. |
Suggested change: Tighten browser alias and safety-mode skill routing. | gstack_boundary_tuned_skill_descriptions scored 0.992. |
| Supabase | terse_supabase_database_mcp scored 0.222. |
Suggested change: Route schema-changing SQL to apply_migration, not execute_sql. |
tuned_supabase_database_boundaries scored 1.000. |
| InsForge | readme_insforge_mcp scored 0.938. |
Suggested change: Reject relative deploy paths before create-deployment. |
source_tuned_insforge_mcp scored 1.000. |
| Screenpipe | readme_screenpipe_mcp scored 0.857. |
Suggested change: Route exact phrase lookup to keyword-search, not broad content search. |
source_tuned_screenpipe_mcp scored 1.000. |
| Firecrawl | legacy_firecrawl_mcp scored 0.000. |
Suggested change: Use firecrawl_scrape for one known page. Reserve firecrawl_extract for broader extraction jobs. |
tuned_firecrawl_mcp_boundaries scored 1.000. |
| Zymtrace | stock_zymtrace_mcp scored 0.583. |
Suggested change: Tighten default-project, resource-first, metrics-first, and bounded hot-trace routing. | tuned_zymtrace_mcp_boundaries scored 1.000. |
| Humwork | readme_humwork_mcp scored 1.000. |
No suggested wording change from this slice. No upstream change is promoted. | skill_tuned_humwork_mcp also scored 1.000. Keep cases as guardrail coverage. |
| OpenWork | docs_openwork_ui_mcp scored 1.000. |
No suggested wording change from this slice. No upstream change is promoted. | source_tuned_openwork_ui_mcp also scored 1.000. Keep cases as guardrail coverage. |
LLM / Machine-readable details
claude_agent_harness_opt/
prompt_builder.py # recipe validation and system prompt rendering
suitability.py # agent task-fit scoring
evals.py # offline answer, tool-use, and final-state evals
trace_review.py # ordered trace review for tools and reasoning
trace_suite.py # regression suites for repeated trace review
agent_audit.py # review tools and traces in one bundle
claude_judge.py # optional Claude Messages API judge for semantic trace review
model_matrix.py # live provider matrix for tool and instruction tuning
harness_optimizer.py # hill-climb candidate tool descriptions from matrix failures
import_run.py # convert external harness exports into audit bundles
live_harness.py # run real harness CLIs and normalize redacted trace artifacts
snapshots.py # pin tool, matrix, skill, and file versions under eval
e2e.py # read-oriented credentialed service and harness checks
reports.py # HTML and PR-comment rendering for JSON results
pr_packets.py # upstream PR packets with pins, examples, and reproduction commands
harness_checks.py # reusable harness optimization check catalog
tool_selection.py # tool description and selection optimizer
value_bar.py # adversarially-confirmed value-bar checks
adapters.py # transcript normalizers for provider and runtime event exports
cli.py # render, score, lint-tools, eval, judge-prompt
recipes/ # ready-to-edit agent recipes
evals/examples/ # small local eval cases
evals/checks/ # reusable harness optimization check families
evals/model_matrix/ # cross-provider model matrix configs
prompts/ # reusable prompt snippets
docs/ # technique map and source map
tests/ # standard-library unit tests
scripts/ # prose gate for public artifacts
.claude/skills/ # project-local Claude Code skill for agent audits
These links are meant to be shared directly. A confirmed packet has a live baseline-to-tuned delta. A guardrail packet means the public catalog already passed the tested slice, so no change is promoted.
scripts/check_finding_packets.py keeps these packet links honest. It verifies each packet has the
required result, evidence, and reproduction sections, is listed in the index and confirmed ledger,
points only at local evidence artifacts that still exist, and validates committed PR packet
evidence.json files against their promoted live matrix results. It also validates committed
evals/results receipts so JSON and Markdown evidence stays structured enough to rerun as eval
fixtures later, reconciles Markdown live-matrix result tables with raw-matrix and optimization-gate
pass/fail, failure, error, skipped, score, result-row identity, and cell-summary tables, requires
unique coverage Markdown summaries backed by sibling JSON receipts, retained coverage gap bullets,
plus recognized unique matrix-summary, tool, and check-family tables when JSON evidence backs them, checks matrix-coverage receipt structure,
gap buckets, and counts, verifies model-matrix receipt rows and strict integer cell summaries plus top-level result summaries against their retained
result rows and source matrices, verifies bounded result-row statuses and status/pass consistency,
checks strict integer aggregate coverage-suite audits and per-matrix summary tables against their
retained source matrices, checks retained live-harness receipt reproduction commands against their claimed source
specs, verifies live-harness receipt cells, typed evidence fields, and status summaries still match
the retained source spec, and audits every retained matrix surface under evals/model_matrix plus matrix-shaped targets
under evals/targets.
scripts/check_eval_surfaces.py keeps the other eval fixtures honest. It validates every retained
example fixture, dry-runs every read-only E2E spec, dry-runs every live harness spec without
credentials, validates live-harness command arrays, version commands, expected tool contracts, and
local script path references, runs every trace suite, and validates the harness-check catalogs.
scripts/check_prompt_recipe_surfaces.py keeps reusable prompt and recipe assets retained as eval
surfaces too. It renders every recipe, lints the recipe tool boundaries, checks use-case suitability,
requires the core operating-loop and value-bar sections in the rendered prompt, and fails new prompt
templates unless they have an explicit surface contract.
scripts/check_skill_surfaces.py applies the same fail-closed rule to project-local skills. It
validates each SKILL.md frontmatter block, routing and reporting sections, referenced CLI
commands, What-To-Look-For categories, and agent metadata under .claude/skills/*/agents.
scripts/check_command_surfaces.py keeps the executable surface synchronized with the docs, project
instructions, retained demo transcript, retained PR packet reproduction commands, and CI.
It verifies every check gate is run in CI and listed below, every check gate has a unit-test file,
every documented claude_agent_harness_opt command names a real CLI subcommand, and command
examples that point at repo fixtures still point at existing files. It derives each subcommand's
valid flags from --help, so documented options cannot drift from the executable parser. Runnable
CLI examples are parse-checked against the real argparse parser, so required arguments and required
options cannot silently disappear from docs. It also
validates documented python scripts/... helper invocations so utility-script examples cannot point
at missing scripts, stale helper options, missing required helper arguments, or stale repo paths. The
helper check distinguishes boolean flags from value-consuming flags when counting required
positionals, validates documented values for argparse choices, and checks simple int or float
typed helper values.
scripts/check_ci_surface.py protects the GitHub Actions contract. It requires push and PR
triggers, read-only permissions, pinned actions, Python 3.11, compile and unit-test smoke checks,
strict coverage jobs, the live Claude judge step, and the no-key negative assertions.
scripts/check_secret_hygiene.py protects the public artifact set. It scans tracked files for
private-key blocks and provider token patterns, verifies .env remains ignored, requires the masked
.env.example, rejects duplicate sample keys, and requires credential-like sample values to stay
blank or placeholder-only.
scripts/check_local_config.py protects local setup and credential docs. It derives required env
keys from E2E specs, model matrices, and probe scripts, then checks .env.example,
docs/setup.md, and docs/credentialed-service-probes.md stay synchronized.
scripts/check_surface_inventory.py protects the full retained-surface crosswalk. It validates
docs/surface-inventory.md, requires every surface family to name owned paths, gates, and
regression material, checks every discovered gate script is listed, and fails when a new evals/*
root or tracked file family is not represented.
scripts/check_regression_ownership.py protects source-to-test ownership. It requires every package
module and script to have a direct regression test or a named wrapper-owner test with import, path,
or CLI evidence, so new source files cannot rely only on broad test discovery.
scripts/check_docs_navigation.py protects the public navigation surface. It verifies repo-local
GitHub links point at existing files or folders, every docs page is reachable from the README,
README layout entries still exist, Makefile help covers public targets, and the package console
script target imports cleanly from pyproject.toml.
scripts/check_source_map.py protects source evidence. It requires docs/source-map.md to keep a
checked date, source sections, README source links, and every external source URL cited by public
docs outside fenced examples.
scripts/check_public_links.py keeps README and docs links shareable. It rejects empty Markdown
targets and local relative links in public docs, including image links, while ignoring examples
inside fenced code blocks.
scripts/check_human_docs.py keeps public Markdown human-first. It requires root docs to put demo,
share links, quickstart, and bundle links before machine-readable detail, requires sendable findings
and PR packets to include the founder handoff, local-agent CTA, and evidence bundle in that order,
and requires machine-heavy docs to move checker, eval, schema, and receipt detail into the bottom
LLM disclosure.
scripts/check_artifact_surfaces.py protects tracked non-code artifacts. It validates
demo.gif against demo.tape
and scripts/render_demo_gif.py,
rejects transcript-reader or secret-bearing demo commands, requires the demo to stay public from the
README, and makes sure committed result receipts are reachable from public docs or PR packet text.
scripts/check_artifact_format.py protects generic tracked artifact formatting. It validates every
tracked JSON file parses, every JSONL line parses, and text artifacts stay nonempty, UTF-8, LF-only,
and newline-terminated.
scripts/check_optimize_shortcuts.py protects the public MCP shortcut runner. It validates every
make optimize mcp=... target in scripts/optimize_mcp.py against its stored matrix, variants,
instruction rules, provider and harness defaults, public docs, Makefile help, and dry selected cells.
scripts/check_makefile_surface.py protects the Makefile command contract directly. It verifies the
public phony targets, selector guardrails, live versus dry-run flag separation, pass-through options,
registered MCP shortcut coverage, documented make ... examples, and a real make help smoke run.
scripts/check_cli_coverage.py protects the executable CLI surface in CI. It parses the public
subcommands from python -m claude_agent_harness_opt --help and requires every one to have a direct
smoke invocation in .github/workflows/ci.yml.
| Topic | Doc |
|---|---|
| GitHub MCP baseline | GitHub MCP Tool Tuning |
| Firecrawl finding | Firecrawl MCP Tool Tuning |
| Supabase finding | Supabase MCP Tool Tuning |
| Screenpipe finding | Screenpipe MCP Tool Tuning |
| InsForge finding | InsForge MCP Tool Tuning |
| Tool-selection optimizer | Tool Selection Optimization |
| Trace review | Trace Review |
| Model matrix | Model Matrix |
| Upstream PR packets | Upstream PR Flywheel |
| Failure discovery and optimization | Hill Descending And Climbing |
| Legacy hill-climb link | Moved: Hill Descending And Climbing |
| Harness imports | Repeatable Harness Lab |
| Live CLI harnesses | Live Harness Hardening |
| SDK harnesses | SDK Harness Coverage |
| Agent SDK reference | Agent SDK Harnesses |
| Service probes | Credentialed Service Probes |
| gstack routing audit | gstack Skill Routing Audit |
| Model migration harnesses | Codex and Model Migration Harnesses |
| Surface inventory | Surface Inventory |
| Optional workflow mapping | Optional Workflow Integration Note |
The repo includes a project-local /agent-audit skill at
.claude/skills/agent-audit/SKILL.md. Use it when reviewing another agent's tools, traces, Claude
Messages API blocks, Agent SDK event exports, IDE-agent event exports, or trace suites. It chooses
the right CLI path, runs deterministic checks, and reports concrete prompt or tool changes. The skill
treats missing value-bar proof as a failed audit.
Use the Claude judge for real audits. It is the semantic gate for reasoning quality, tool-output use, and final grounding:
export ANTHROPIC_API_KEY=...
python -m claude_agent_harness_opt review-trace evals/examples/agent_trace_good.json --claude-judge
python -m claude_agent_harness_opt audit-agent evals/examples/agent_audit_bundle.json --claude-judge --markdownThe judge reviews only visible artifacts: reasoning summaries, provider-returned reasoning blocks when available, tool calls, tool outputs, and final answers. It does not claim access to hidden chain-of-thought. If an agent runtime does not expose reasoning, instrument the agent to emit short decision notes before and after tool calls.
GitHub Actions requires the ANTHROPIC_API_KEY repository secret and runs the same live judge
against the sample audit bundle on every push and pull request.
Use optimize-tools when the question is whether the agent has the right tool descriptions, schemas,
and calibration cases to choose tools correctly:
python -m claude_agent_harness_opt optimize-tools evals/examples/tool_tuning_before_bundle.json --markdown || true
python -m claude_agent_harness_opt optimize-tools evals/examples/agent_audit_bundle.json --markdown
python -m claude_agent_harness_opt optimize-tools evals/examples/agent_audit_bundle.json --claude-judgeThe first command is a deliberate before-state negative control. It should fail and recommend an
input_schema, calibration cases, held-out cases, stronger avoid guidance, quality checks, and
runtime metrics. The second command is the after-state sample that passes.
The public MCP shortcut wraps the stored matrices for known targets:
make optimize mcp=insforge
make optimize mcp=humwork
make optimize mcp=openwork
make optimize mcp=screenpipe
make optimize mcp=firecrawl
make optimize mcp=supabase
make optimize mcp=zymtrace
make optimize mcp=clickhouse
make optimize mcp=github
make optimize mcp=context7
make optimize url=https://github.com/InsForge/insforge-mcpUnknown URLs fail rather than falling back to another target.
The optimizer checks that every tool has a distinct purpose, use_when, avoid_when,
input_schema, and result quality_checks. It also checks tool_selection_cases and maps trace
failures back to concrete changes like stronger avoid rules, argument schemas, examples, or stop
criteria. audit-agent --claude-judge includes this optimizer automatically.
The process is a bounded search, not a random prompt sweep.
Start by inventorying the surface under test:
- public docs and source pins
- MCP
tools/listand resource list - installed skills and project rules
- live smoke calls when credentials are available
- existing traces or user reports that show bad tool choice
Then build a boundary map. A good boundary is one where the first wrong tool call changes cost, safety, correctness, or recoverability. Common boundaries are:
- adjacent tools that sound similar
- required arguments that models often omit
- resource-first paths versus direct tool fallback
- broad search versus exact lookup
- discovery calls versus value-query calls
- metadata discovery versus full payload fetch
- no-tool safety cases where any tool call is wrong
- workflow rules supplied by a skill or
CLAUDE.md
Skills are one input, not the whole descent. They are useful because they encode expert workflow rules, but the descent also uses live MCP inventories, generated schemas, resource lists, upstream docs, source pins, trace failures, support reports, smoke-call output, and existing result receipts. A case is stronger when at least two of those sources agree that a boundary matters.
Completeness comes from forcing every retained boundary through the same loop: it needs a source reference, a matrix or fixture case, a deterministic gate, and a retained result or packet. Skills seed workflow rules. MCP schemas seed argument coverage. Docs and source pins seed intended use. Traces and support reports seed known failure modes. CI and Makefile commands seed executable entrypoints. If a surface cannot be represented in one of those durable forms, treat it as an unproven observation rather than coverage.
The coverage target is the product of those inputs, not a single list of skills. For completeness, build cases across these axes:
- skill workflow rules and
CLAUDE.mdstyle host rules - live MCP tool names, descriptions, and argument schemas
- MCP resources and the resource-first versus direct-tool fallback path
- generated REST helpers and low-level tools that the model may overuse
- exact lookup, broad search, metadata, drilldown, write, and no-tool safety families
- provider and harness variants, including native tools and prompt-JSON wrappers
- known trace failures, support reports, and smoke-call observations
That is the "hill descending" part. We deliberately walk toward likely failure valleys by writing
adversarial prompts that make one boundary measurable. Each case names the expected tool,
confusable alternatives, a check_family, and any required argument checks. The point is to find a
small prompt where the transcript can be judged without taste.
Run a corpus audit before claiming the matrix is broad enough:
python -m claude_agent_harness_opt matrix-coverage evals/model_matrix/zymtrace_mcp_tool_selection.json --markdown
python -m claude_agent_harness_opt matrix-coverage-suite evals/model_matrix evals/targets/gstack/gstack_skill_selection_matrix.json --markdownmatrix-coverage checks the matrix as an eval corpus before any provider call:
- every catalog tool is expected by at least one case
- every catalog tool appears as a forbidden confusable somewhere
- expected tools with arguments have at least one argument assertion
- every tuned tool has quality checks
- every case has forbidden tools and a
check_family - every required
check_familydeclared by the matrix is covered - every tool-description variant exposes the same tool surface unless the matrix explicitly opts out
- duplicate tool names inside a variant are surfaced
source.tool_countmatches the effective tool surface when it is declared- case, profile, tool-variant, and instruction-variant identities are present and unambiguous
- matrix case, profile, tool-variant, instruction-variant, harness, and tool lists have valid shape
- case expectations do not duplicate tools, overlap expected and forbidden tools, or mix no-tool safety with expected operational tools
- argument assertions reference arguments exposed by at least one expected tool schema across the matrix variants
- expected and forbidden tool fields have the shape the runner will evaluate
- unknown expected or forbidden tool names fail the audit
- any matrix
value_barmetadata clears the same adversarial-confirmation gate used by the repo
This does not replace live scoring. It prevents a clean live run from hiding an untested tool,
untested negative, missing argument boundary, forgotten family, accidental tool-surface drift, typoed
tool reference, contradictory case expectation, partial value-bar evidence, or ambiguous fixture
identity. The coverage.required_check_families field is the edge-family contract for a matrix.
coverage.allow_variant_tool_delta is reserved for matrices that intentionally compare different
tool catalogs. Store the matrix, coverage report, live result, and PR packet together so the same
cases can be rerun later as evals. These artifacts are not scratch notes: the repository checks
validate result receipt shape, matrix paths, coverage summaries, baseline failure rows, promoted
candidate evidence, model-matrix result and cell summaries, optimization-gate pass/fail,
skipped/error counts, JSON result-row status vocabulary and status/pass consistency, Markdown
result-row shape and identity, Markdown cell summaries, unique required coverage summary lines, matrix-coverage receipt item and gap-bucket structure, complete coverage gap labels and recognized unique required coverage tables, coverage-suite
matrix rows and strict integer aggregate counts, live-harness cell fields and strict integer status summaries, and Markdown review sections so future agents can use
them as regression fixtures.
For the full repository, the current ledger is stored at
evals/results/model_matrix_coverage_suite_2026-06-30.md: it audits 19 matrices, 182 tools, 230
cases, 57 profile surfaces, 23 instruction variants, 866 boundary pairs, and 4 value bars. All stored
model matrices plus the gstack target matrix now pass the strict structural coverage contract with
zero case expectation gaps, zero identity gaps, and zero value-bar gaps. That proves catalog
coverage, negative coverage, argument assertions, quality checks, and family labels are present,
that each matrix's required family contract is covered, that argument checks map to expected tool
schemas, that value-bar metadata is complete where present, and that variant tool surfaces have
parity. It does not prove every live model will choose correctly, so promoted behavioral claims
still need live model-matrix results.
After baseline failures repeat, the "hill climb" part starts:
- Run
model-matrixon the baseline tool surface. - Cluster failures by tool, argument, provider, harness, and instruction variant.
- Draft the smallest candidate change that explains those failures.
- Rerun the same live cells against baseline and candidate.
- Rerun held-out cases that should not have been tailored to the failure.
- Promote only when the candidate beats the threshold and held-out cells do not regress.
- Store the result JSON and upstream packet so another maintainer can rerun or challenge it.
grind-harness automates that climb for simple description and harness variants:
python -m claude_agent_harness_opt grind-harness evals/model_matrix/coding_tool_selection.json \
--env-file .env \
--live \
--require-live \
--providers anthropic,openai,gemini \
--harnesses native_tools,prompt_json \
--instruction-variants boundary_rules \
--cases "investigate trace review flow,map model matrix implementation" \
--heldout-cases "find python files,read known file" \
--min-improvement 0.05 \
--max-live-calls 80 \
--concurrency 8 \
--markdownFor richer MCP surfaces, the same loop is often run manually first because the fix may span tool descriptions, resource guidance, generated OpenAPI schemas, and skill instructions.
Zymtrace is the concrete example. The inspected MCP surface exposed 25 tools and 3 resources, while
the installed skills added CPU, GPU, allocation, resource-first, default-project, and bounded
hot_traces rules. That produced boundary cases such as:
- use default project
00000000-0000-0000-0000-000000000000instead of searching projects - discover metrics with
project_metrics_activity_aggrbefore querying values - start GPU and inference investigations with GPU, CPU, and framework metrics
- use
topentitiesortopfunctionsfor rank-first CPU requests - use MCP resources first for
topfunctions,topentities, andflamegraph - use
hot_tracesmetadata first withmeta_only=trueand a small limit - fetch a full trace only with a selected
prefix_hash,meta_only=false, andlimit=1
The first Zymtrace matrix proved an improvement but the coverage audit found untouched generated
REST helpers. The hardened matrix now has 34 cases, covers 25 of 25 tools as expected tools, covers
25 of 25 tools as forbidden confusables, carries 85 boundary pairs, and labels every case with a
check_family.
The expanded Zymtrace live run used those boundaries across Anthropic, OpenAI, and Gemini:
python -m claude_agent_harness_opt model-matrix evals/model_matrix/zymtrace_mcp_tool_selection.json \
--env-file .env \
--live \
--require-live \
--providers anthropic,openai,gemini \
--harnesses prompt_json \
--variants stock_zymtrace_mcp,tuned_zymtrace_mcp_boundaries \
--instruction-variants zymtrace_host_and_skill_rules \
--cases "default project metrics discovery skips search,cpu rank first containerized apps,gpu inference workflow starts with metrics,gpu call tree uses hot traces,selected trace drilldown is bounded,full trace error recovers to discovery,hot trace discovery is bounded,resource fallback hot functions" \
--concurrency 3 \
--out /tmp/zymtrace-live.jsonThe stock Zymtrace surface passed 14 of 24 selected cells. The tuned surface passed 24 of 24. That is why the Zymtrace finding was promoted and packaged under docs/findings/zymtrace.
These are shortened examples of the kind of change the loop promotes. The full packets live under docs/findings.
Firecrawl fixed a scrape-versus-extract boundary:
- Use firecrawl_extract for structured fields from a page.
+ Use firecrawl_scrape when the exact page URL is known and the task needs that page's content,
+ metadata, screenshot, branding, or structured fields.
+ Use firecrawl_extract when the user asks for fields across several pages or a broader structured
+ extraction job. Avoid for one known URL.Supabase fixed a migration-safety boundary:
- Use execute_sql for SQL statements.
+ Use apply_migration for DDL and schema-changing SQL.
+ Use execute_sql only for regular SQL that does not change database schema.Zymtrace fixed workflow and argument boundaries:
- Use project tools for the requested project.
- Use hot_traces for call-tree analysis.
+ Use default project 00000000-0000-0000-0000-000000000000 unless the user names another project.
+ Discover metrics with project_metrics_activity_aggr before querying metric values.
+ First hot_traces calls use meta_only=true and limit<=5.
+ Full trace fetches require prefix_hash, meta_only=false, and limit=1.Screenpipe fixed exact keyword lookup:
- Use search-content for screen or transcript search.
+ Use keyword-search for literal terms and exact phrases.
+ Use search-content for transcript lines, screen text, speaker or window filters, tags, memories,
+ and broader content search.InsForge fixed a no-tool safety case:
- Use create-deployment when the user asks to deploy a local source tree.
+ Use create-deployment only when the user provides an absolute sourceDirectory path.
+ Return NO_TOOL for relative paths like "." until the path is resolved.Use model-matrix when tuning tool descriptions or CLAUDE.md style instructions for a new model,
provider, reasoning mode, or harness:
python -m claude_agent_harness_opt model-matrix evals/model_matrix/coding_tool_selection.json --markdown
python -m claude_agent_harness_opt matrix-coverage evals/model_matrix/zymtrace_mcp_tool_selection.json --markdown
python -m claude_agent_harness_opt matrix-coverage-suite evals/model_matrix evals/targets/gstack/gstack_skill_selection_matrix.json --markdown --out /tmp/model-matrix-coverage.md
python -m claude_agent_harness_opt model-matrix evals/model_matrix/coding_tool_selection.json --env-file .env --live --concurrency 8 --markdown
python -m claude_agent_harness_opt model-matrix evals/model_matrix/agent_audit_skill_selection.json --env-file .env --live --require-live --providers anthropic --harnesses prompt_json --variants thin_workflow_tools --instruction-variants no_skill,agent_audit_skill --markdown
python -m claude_agent_harness_opt model-matrix evals/model_matrix/github_mcp_tool_selection.json --env-file .env --live --require-live --providers anthropic,openai,gemini --harnesses native_tools,prompt_json --variants stock_github_mcp,tuned_github_mcp_boundaries --instruction-variants github_mcp_host_rules --concurrency 4 --markdown
make optimize mcp=screenpipe
make optimize url=https://github.com/screenpipe/screenpipemodel-matrix and grind-harness fail when filters select zero cells. A typoed provider, harness,
tool variant, instruction variant, or case name is not a passing empty run. The error prints the
requested values and the available matrix values so CI logs point to the missing surface directly.
The included matrix tests Claude Code style Task, Glob, Grep, and Read tool selection across
Anthropic, OpenAI, and Gemini. It compares short descriptions against tuned boundary descriptions,
and it compares native provider tool calling against a standardized JSON-choice harness.
The agent-audit skill matrix shows how to test a skill as an instruction variant against a no-skill
baseline and a thin-description stress case.
The GitHub MCP matrix shows how to import a public MCP tool catalog, compare stock descriptions to
a tuned boundary variant, and avoid promoting description changes when the stock catalog already
passes.
Use grind-harness when the goal is to tune the harness itself. It runs a baseline matrix cell,
creates a candidate tool-description variant from the failed cases, reruns the selected cells,
checks held-out cells, and marks the value bar as passed only when the live candidate beats the
baseline by the configured threshold without regressions:
python -m claude_agent_harness_opt grind-harness evals/model_matrix/coding_tool_selection.json \
--env-file .env \
--live \
--require-live \
--providers anthropic,openai,gemini \
--harnesses native_tools,prompt_json \
--instruction-variants boundary_rules \
--cases "investigate trace review flow,map model matrix implementation" \
--heldout-cases "find python files,read known file" \
--min-improvement 0.05 \
--concurrency 8 \
--markdownTreat each runtime as a harness target. Provider native tools, prompt JSON wrappers, Agent SDK loops, Codex JSONL exports, IDE agents, and Cursor-like environments should export the same visible trace contract: decision notes, tool calls, tool results, and final answers. Once the adapter emits that contract, the trace suite, Claude judge, model matrix, and harness grind can compare it against other harnesses. See Harness Optimization for the adapter and upgrade loop.
To test an exported harness without an API key, normalize a runtime event file and run the fixture matrix:
python -m claude_agent_harness_opt normalize-runtime evals/examples/agent_sdk_trace_review_events.json
python -m claude_agent_harness_opt model-matrix evals/model_matrix/harness_trace_adapters.json \
--live \
--require-live \
--providers trace_fixture \
--harnesses agent_sdk_trace,cursor_trace \
--variants exported_trace_tools \
--instruction-variants exported_trace \
--markdown
python -m claude_agent_harness_opt model-matrix evals/model_matrix/codex_harness_trace_adapter.json \
--live \
--require-live \
--providers trace_fixture \
--harnesses codex_exec_jsonl \
--variants codex_exported_trace_tools \
--instruction-variants codex_exported_trace \
--markdownpython scripts/deslop_check.py
python -m compileall claude_agent_harness_opt scripts
python -m unittest discover -s tests -q
python scripts/check_prompt_recipe_surfaces.py
python scripts/check_skill_surfaces.py
python scripts/check_command_surfaces.py
python scripts/check_ci_surface.py
python scripts/check_secret_hygiene.py
python scripts/check_local_config.py
python scripts/check_surface_inventory.py
python scripts/check_regression_ownership.py
python scripts/check_docs_navigation.py
python scripts/check_source_map.py
python scripts/check_public_links.py
python scripts/check_human_docs.py
python scripts/check_artifact_surfaces.py
python scripts/check_artifact_format.py
python scripts/check_makefile_surface.py
python scripts/check_optimize_shortcuts.py
python scripts/check_cli_coverage.py
python scripts/check_project_instructions.py
python scripts/check_package_surface.py
python -m claude_agent_harness_opt judge-prompt evals/examples/search_answer.json > /tmp/judge-prompt.txt
python -m claude_agent_harness_opt eval evals/examples/search_answer.json
python -m claude_agent_harness_opt review-trace evals/examples/agent_trace_good.json
python -m claude_agent_harness_opt normalize-claude evals/examples/claude_messages.json
python -m claude_agent_harness_opt normalize-runtime evals/examples/cursor_trace_review_events.json
python -m claude_agent_harness_opt import-run evals/examples/import_run_cursor_export.json --adapter cursor --out-dir /tmp/imported-run
python -m claude_agent_harness_opt audit-agent /tmp/imported-run/agent_audit_bundle.json
python -m claude_agent_harness_opt snapshot-surface --matrix evals/model_matrix/harness_trace_adapters.json --bundle evals/examples/agent_audit_bundle.json --skill .claude/skills/agent-audit/SKILL.md --out /tmp/surface-snapshot.json
python scripts/check_eval_surfaces.py
python -m claude_agent_harness_opt trace-suite evals/suites/agent_trace_suite.json
python -m claude_agent_harness_opt audit-agent evals/examples/agent_audit_bundle.json
python -m claude_agent_harness_opt audit-agent evals/examples/agent_audit_bundle.json --claude-judge
python -m claude_agent_harness_opt optimize-tools evals/examples/agent_audit_bundle.json --claude-judge
python -m claude_agent_harness_opt model-matrix evals/model_matrix/coding_tool_selection.json
python -m claude_agent_harness_opt model-matrix evals/model_matrix/agent_audit_skill_selection.json --providers anthropic --harnesses prompt_json --variants thin_workflow_tools --instruction-variants agent_audit_skill --max-cases 2
python -m claude_agent_harness_opt model-matrix evals/model_matrix/github_mcp_tool_selection.json --providers anthropic --harnesses prompt_json --variants stock_github_mcp --instruction-variants github_mcp_host_rules --max-cases 2
python -m claude_agent_harness_opt model-matrix evals/model_matrix/firecrawl_mcp_tool_selection.json --providers anthropic --harnesses prompt_json --variants tuned_firecrawl_mcp_boundaries --instruction-variants firecrawl_host_rules --max-cases 2
python -m claude_agent_harness_opt model-matrix evals/model_matrix/supabase_mcp_database_tool_selection.json --providers anthropic --harnesses prompt_json --variants tuned_supabase_database_boundaries --instruction-variants supabase_database_host_rules --max-cases 2
python -m claude_agent_harness_opt model-matrix evals/model_matrix/clickhouse_mcp_tool_selection.json --providers anthropic --harnesses prompt_json --variants tuned_clickhouse_readonly_boundaries --instruction-variants clickhouse_host_rules --max-cases 2
python -m claude_agent_harness_opt matrix-coverage evals/model_matrix/zymtrace_mcp_tool_selection.json --strict --out /tmp/zymtrace-coverage.json
python -m claude_agent_harness_opt matrix-coverage-suite evals/model_matrix evals/targets/gstack/gstack_skill_selection_matrix.json --out /tmp/model-matrix-coverage-suite.json
python -m claude_agent_harness_opt model-matrix evals/model_matrix/zymtrace_mcp_tool_selection.json --providers anthropic --harnesses prompt_json --variants tuned_zymtrace_mcp_boundaries --instruction-variants zymtrace_host_and_skill_rules --max-cases 2
python -m claude_agent_harness_opt model-matrix evals/model_matrix/harness_trace_adapters.json --live --require-live --providers trace_fixture
python -m claude_agent_harness_opt model-matrix evals/model_matrix/codex_harness_trace_adapter.json --live --require-live --providers trace_fixture
python -m claude_agent_harness_opt model-matrix evals/model_matrix/harness_trace_adapters.json --providers trace_fixture --harnesses agent_sdk_trace --max-cases 1 --out /tmp/harness-matrix.json
python -m claude_agent_harness_opt render-report /tmp/harness-matrix.json --out /tmp/harness-matrix.html
python -m claude_agent_harness_opt pr-comment /tmp/harness-matrix.json --out /tmp/harness-matrix.md
python -m claude_agent_harness_opt harness-checks --markdown
python -m claude_agent_harness_opt upstream-pr-packet /tmp/harness-matrix.json --target-name "Example MCP" --baseline-variant stock --candidate-variant tuned --out-dir /tmp/upstream-pr
python -m claude_agent_harness_opt grind-harness evals/model_matrix/coding_tool_selection.json
python scripts/probe_service_keys.py --env-file .env --no-fail
python scripts/check_finding_packets.py
python scripts/check_value_bar.pyThe technique map is grounded in Anthropic's public video and docs. See Source Map for the source list and timestamps used while building the repo. See Video Coverage Audit for the implementation coverage check against the talk.
MIT.