Validation run: GlobalOpinionQA × Claude Haiku 4.5 (direct OpenRouter)

[the-data-viking]

Replicate the published GlobalOpinionQA run for openrouter/anthropic/claude-haiku-4-5. The dataset auto-downloads from its public source (Anthropic/llm_global_opinions on Hugging Face) — no manual setup.

See #337 for what validation runs are, why they matter, and the report template to post here when done.

Licensing note: GlobalOpinionQA is CC-BY-NC-SA; SynthBench treats it as a gated dataset for republication, meaning per-question human_distribution values must not land in this public repo. Submit via the web/CLI path (the server handles stripping and canonical rehydration automatically). If you insist on the PR path, run python scripts/strip-gated-distributions.py on your file first — CI rejects unstripped files.

spec:
  task: validation-run
  dataset: globalopinionqa
  provider: openrouter
  model: anthropic/claude-haiku-4-5
  samples_per_question: 15
  n_questions: 100             # pass -n 100
  temperature: null            # provider default; do not pass --temperature
  refusal_detector_version: 2
  reference_file: globalopinionqa_openrouter_anthropic_claude-haiku-4-5_20260716_190818.json
  reference_scores:
    sps: 0.679
    sps_ci95: [0.638, 0.716]
    p_dist: 0.671
    p_rank: 0.616
    p_refuse: 0.750
    parse_failure_rate: 0.131
  reference_hashes:
    question_set: 4f27a038b9f8a71d9b4cc9a3c5f41b62dcdd4238146c9ca23729d2364e3aaf28
    prompt_template: "sha256:2b0274a2ec39cc1458454fc0d3af179c659a37a1ddd38a1c136250c22d9bc1be"
  success_criterion: "recomputed sps within [0.638, 0.716], or overlapping 95% CIs"
  estimated_cost_usd: 1
  estimated_runtime_min: 20   # at default concurrency (-c 10); raise -c to go faster

Steps

The PyPI distribution is synthbench-eval (the bare synthbench name on PyPI belongs to an unrelated project). It installs the synthbench CLI.

# 0. Install.
python3 -m venv .venv && source .venv/bin/activate
pip install "synthbench-eval[openai]>=0.5.0"

# 1. Run the benchmark (dataset downloads automatically on first run).
export OPENROUTER_API_KEY=...
synthbench run -p openrouter -m anthropic/claude-haiku-4-5 -d globalopinionqa -s 15 -n 100 -o results/

# 2. Validate:
synthbench validate --tier3 \
  --expected-question-hash 4f27a038b9f8a71d9b4cc9a3c5f41b62dcdd4238146c9ca23729d2364e3aaf28 \
  results/<your-file>.json

# 3. Submit (agents: see #337 → “Getting submission access” — your operator mints a submit-scope key once at https://synthbench.org/account/, or use the PR path):
export SYNTHBENCH_API_KEY=sb_...
synthbench submit results/<your-file>.json

Notes:

• A parse-failure rate near the reference's 13% is expected on this target and should be reported, not "fixed" — silently dropping unparseable answers is exactly what the validator's plausibility checks look for.
• Short-answer warnings from tier-3 (RAW_RESPONSES_LENGTH_SHORT) are benign on multiple-choice targets.

Then post the report template from #337 as a comment here.