Skip to content

refactor: Benchmark should compare models, not prompt strategies #2

Description

@sangicook

Context

The live benchmark (run_live_benchmark.py) currently runs a 2-arm experiment comparing Typed vs Raw prompts on the same model (Gemini Flash). We already know Typed is strictly better — Raw prompts invent YAML paths and fail consistently.

The benchmark should instead compare models on the same (Typed) prompt strategy.

Current State

configs = [
    BenchmarkConfig(prompt_builder=build_typed_action_prompt, model="gemini-flash", label="Typed Gemini Flash"),
    BenchmarkConfig(prompt_builder=build_consultation_prompt, model="gemini-flash", label="Raw Gemini Flash (control)"),
]

Proposed Change

configs = [
    BenchmarkConfig(prompt_builder=build_typed_action_prompt, model="gemini-flash", label="Gemini Flash"),
    BenchmarkConfig(prompt_builder=build_typed_action_prompt, model="gpt-4.1-mini", label="GPT-4.1 Mini"),
    # etc.
]

Key decisions

  • Drop the Raw arm entirelybuild_consultation_prompt and collect_agent_proposals (raw pipeline) can be deprecated
  • All arms use Typed actionsbuild_typed_action_prompt + collect_typed_proposals only
  • Expand MODEL_REGISTRY with candidate models to benchmark
  • Report should rank models by resolution rate, CQS lift, cost, and cop-out rate
  • Cost-per-kept-change becomes the key efficiency metric

Notes

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions