Validation run: GSS × Claude Haiku 4.5 via SynthPanel personas (current GSS #1)

[the-data-viking]

Replicate the top-ranked GSS entry, which runs Claude Haiku 4.5 through the SynthPanel persona pipeline rather than direct prompting. This is the highest-value target on the board: it's both the #1 GSS score and an operator-adjacent one (SynthPanel is our sibling project), so independent confirmation matters most here.

See #337 for what validation runs are, why they matter, and the report template to post here when done.

spec:
  task: validation-run
  dataset: gss
  provider: synthpanel
  model: openrouter/anthropic/claude-haiku-4-5
  samples_per_question: 30
  n_questions: 75            # all questions; do not pass -n
  temperature: null          # provider default; do not pass --temperature
  refusal_detector_version: 2
  reference_file: "leaderboard-results/gss_synthpanel_openrouter_anthropic_claude-haiku-4-5_20260718_220554.json"
  reference_scores:
    sps: 0.806
    sps_ci95: [0.770, 0.834]
    p_dist: 0.718
    p_rank: 0.729
    p_refuse: 0.972
    parse_failure_rate: 0.015
  reference_hashes:
    question_set: 813eca34cc554502164c08944eccde8ba4360612a333e69c3b39785d451b007b
    prompt_template: "sha256:4f1c561ab4b48967796548fd5fa849c4897cfd87aaca694e6a359e4663546ab6"
  success_criterion: "recomputed sps within [0.770, 0.834], or overlapping 95% CIs"
  estimated_cost_usd: 3
  estimated_runtime_min: 45   # at default concurrency (-c 10); raise -c to go faster

Steps

The PyPI distribution is synthbench-eval (the bare synthbench name on PyPI belongs to an unrelated project); it installs the synthbench CLI. synthpanel on PyPI is ours.

# 0. Install (the clone is only needed for the GSS ingest script).
git clone https://github.com/DataViking-Tech/SynthBench && cd SynthBench
python3 -m venv .venv && source .venv/bin/activate
pip install "synthbench-eval[openai]>=0.5.0" synthpanel pandas pyreadstat

# 1. Build the GSS fixture (public domain, NORC 2024 release; deterministic).
python scripts/ingest-gss.py --data-dir data/gss

# 2. Run the benchmark through the synthpanel persona pipeline.
export OPENROUTER_API_KEY=...
synthbench run -p synthpanel -m openrouter/anthropic/claude-haiku-4-5 -d gss -s 30 \
  --data-dir data/gss -o results/

# 3. Validate:
synthbench validate --tier3 \
  --expected-question-hash 813eca34cc554502164c08944eccde8ba4360612a333e69c3b39785d451b007b \
  results/<your-file>.json

# 4. Submit (agents: see #337 → “Getting submission access” — your operator mints a submit-scope key once at https://synthbench.org/account/, or use the PR path):
export SYNTHBENCH_API_KEY=sb_...
synthbench submit results/<your-file>.json

Notes:

• Check that your output's config.prompt_template_hash equals the reference (sha256:4f1c56…). A synthpanel version with changed persona templates produces a different hash, which makes scores non-comparable — if the hashes differ, report that too (with your synthpanel --version); template drift across releases is itself something we want flagged.

• Reference parse_failure_rate was 0.015; report yours. Validation of your run should produce zero raw-responses warnings — the provider populates raw_responses as of synthbench-eval 0.5.0; if you see RAW_RESPONSES_* issues, check your installed version.

• If scripts/ingest-gss.py times out downloading from NORC (its download has no retry yet), fetch the zip yourself and hand the script the file: download https://gss.norc.org/content/dam/gss/get-the-data/documents/stata/2024_stata.zip, unzip, then re-run with --dta /path/to/GSS2024.dta.

Then post the report template from #337 as a comment here.

[the-data-viking]

Maintainer note — worked example. As with #338, we dry-ran this issue end-to-end with an autonomous AI agent (fresh clone, PyPI install, own venv, no prior project knowledge) before promoting it. It replicated the reference run — and this round also exercised the raw_responses fix shipped in synthbench-eval 0.5.0 (#347). This does not count as a third-party validation (we ran it):

Validation report (internal dry run — not a third-party validation)

• Leaderboard submission ID / link: not submitted (dry run; agent had no synthbench.org account)
• Recomputed sps: 0.801 (95% CI [0.761, 0.828])
• Reference sps: 0.806 (95% CI [0.770, 0.834]) → REPLICATED
• question_set_hash matches reference: yes
• prompt_template_hash matches reference: yes (synthpanel 1.6.0 — no template drift)
• parse_failure_rate: 0.0147 (33/2250 — reference: 0.015)
• synthbench version: synthbench-eval 0.5.0
• Anything unusual: zero validation warnings (tier-3 clean, raw_responses fully populated); NORC's download timed out once (fixed in fix(scripts): retry NORC download in ingest-gss with timeout and fallback hint #348, manual --dta fallback now documented above); actual runtime ~6 min at -c 20 and cost ≈ $1.70.

The issue body has been updated based on this run: install line pinned to >=0.5.0 (an unpinned install minutes after a release can resolve to the previous version while the PyPI index propagates), and the NORC manual-fallback note added.