Validation run: GSS × Gemini 2.5 Flash (direct OpenRouter)

[the-data-viking]

Replicate the published GSS run for openrouter/google/gemini-2.5-flash. This is the easiest validation target: fully public (public-domain) dataset, one API key, roughly $0.10 in API cost.

See #337 for what validation runs are, why they matter, and the report template to post here when done.

spec:
  task: validation-run
  dataset: gss
  provider: openrouter
  model: google/gemini-2.5-flash
  samples_per_question: 30
  n_questions: 75            # all questions; do not pass -n
  temperature: null          # provider default; do not pass --temperature
  reference_file: leaderboard-results/gss_openrouter_google_gemini-2.5-flash_20260715_175807.json
  reference_scores:
    sps: 0.789
    sps_ci95: [0.759, 0.817]
    p_dist: 0.699
    p_rank: 0.697
    p_refuse: 0.972
    mean_jsd: 0.301
    parse_failure_rate: 0.004
  reference_hashes:
    question_set: 813eca34cc554502164c08944eccde8ba4360612a333e69c3b39785d451b007b
    prompt_template: "sha256:2b0274a2ec39cc1458454fc0d3af179c659a37a1ddd38a1c136250c22d9bc1be"
  success_criterion: "recomputed sps within [0.759, 0.817], or overlapping 95% CIs"
  expected_warnings: ["RAW_RESPONSES_LENGTH_SHORT"]   # benign; the reference file trips the same one
  estimated_cost_usd: 0.10
  estimated_runtime_min: 30   # at default concurrency (-c 10); raise -c to go faster

Steps

The PyPI distribution is synthbench-eval (the bare synthbench name on PyPI belongs to an unrelated project). It installs the synthbench CLI.

# 0. Install (the clone is only needed for the GSS ingest script).
git clone https://github.com/DataViking-Tech/SynthBench && cd SynthBench
python3 -m venv .venv && source .venv/bin/activate
pip install "synthbench-eval[openai]>=0.5.0" pandas pyreadstat

# 1. Build the GSS fixture (public domain, NORC 2024 single-year release).
#    Deterministic; defaults (2024, weighted) match the reference run.
#    --data-dir keeps the fixture local instead of in ~/.synthbench.
python scripts/ingest-gss.py --data-dir data/gss

# 2. Run the benchmark (~30 min at default concurrency; ~2,250 calls).
export OPENROUTER_API_KEY=...
synthbench run -p openrouter -m google/gemini-2.5-flash -d gss -s 30 \
  --data-dir data/gss -o results/

# 3. Verify your run is internally valid and the question set matches:
synthbench validate --tier3 \
  --expected-question-hash 813eca34cc554502164c08944eccde8ba4360612a333e69c3b39785d451b007b \
  results/<your-file>.json

# 4. Submit (agents: see #337 → “Getting submission access” — your operator mints a submit-scope key once at https://synthbench.org/account/, or use the PR path):
export SYNTHBENCH_API_KEY=sb_...
synthbench submit results/<your-file>.json

Notes:

• Step 3 will emit a RAW_RESPONSES_LENGTH_SHORT warning — expected on this target (the model answers with single option letters) and not a failure; the published reference file produces the identical warning.

• If question_set_hash in your output doesn't match the reference, your GSS fixture differs — re-run scripts/ingest-gss.py with defaults and check the NORC release year (2024).

• The reference run predates the refusal_detector_version config stamp (current code uses v2); this has been verified not to affect replication on this target.

• If scripts/ingest-gss.py times out downloading from NORC (its download has no retry yet), fetch the zip yourself and hand the script the file: download https://gss.norc.org/content/dam/gss/get-the-data/documents/stata/2024_stata.zip, unzip, then re-run with --dta /path/to/GSS2024.dta.

Then post the report template from #337 as a comment here.

[the-data-viking]

Maintainer note — worked example. Before opening this issue up, we dry-ran it end-to-end with an autonomous AI agent given nothing but the issue text, a fresh clone, and an OpenRouter key. It replicated the reference run. This does not count as a third-party validation (we ran it), but it shows the workflow works and what a completed report looks like:

Validation report (internal dry run — not a third-party validation)

• Leaderboard submission ID / link: not submitted (dry run; agent had no synthbench.org account)
• Recomputed sps: 0.799 (95% CI [0.768, 0.826])
• Reference sps: 0.789 (95% CI [0.759, 0.817]) → REPLICATED
• question_set_hash matches reference: yes
• prompt_template_hash matches reference: yes
• parse_failure_rate: 0.004 (8/2250 — reference: 9/2250)
• synthbench version: 0.1.0 (installed from main)
• Anything unusual: RAW_RESPONSES_LENGTH_SHORT tier-3 warning (benign — the reference file trips the identical warning); actual API cost ≈ $0.08.

The issue body has been updated based on the agent's friction log: install from source instead of PyPI (the synthbench PyPI name is an unrelated package), the required [openai] extra, realistic runtime at default concurrency, --data-dir isolation, and the expected-warnings note above.