Context
The live benchmark (run_live_benchmark.py) currently runs a 2-arm experiment comparing Typed vs Raw prompts on the same model (Gemini Flash). We already know Typed is strictly better — Raw prompts invent YAML paths and fail consistently.
The benchmark should instead compare models on the same (Typed) prompt strategy.
Current State
configs = [
BenchmarkConfig(prompt_builder=build_typed_action_prompt, model="gemini-flash", label="Typed Gemini Flash"),
BenchmarkConfig(prompt_builder=build_consultation_prompt, model="gemini-flash", label="Raw Gemini Flash (control)"),
]
Proposed Change
configs = [
BenchmarkConfig(prompt_builder=build_typed_action_prompt, model="gemini-flash", label="Gemini Flash"),
BenchmarkConfig(prompt_builder=build_typed_action_prompt, model="gpt-4.1-mini", label="GPT-4.1 Mini"),
# etc.
]
Key decisions
- Drop the Raw arm entirely —
build_consultation_prompt and collect_agent_proposals (raw pipeline) can be deprecated
- All arms use Typed actions —
build_typed_action_prompt + collect_typed_proposals only
- Expand
MODEL_REGISTRY with candidate models to benchmark
- Report should rank models by resolution rate, CQS lift, cost, and cop-out rate
- Cost-per-kept-change becomes the key efficiency metric
Notes
Context
The live benchmark (
run_live_benchmark.py) currently runs a 2-arm experiment comparing Typed vs Raw prompts on the same model (Gemini Flash). We already know Typed is strictly better — Raw prompts invent YAML paths and fail consistently.The benchmark should instead compare models on the same (Typed) prompt strategy.
Current State
Proposed Change
Key decisions
build_consultation_promptandcollect_agent_proposals(raw pipeline) can be deprecatedbuild_typed_action_prompt+collect_typed_proposalsonlyMODEL_REGISTRYwith candidate models to benchmarkNotes