Why
Every entry on the synthbench.org leaderboard was produced or triaged by the maintainer team. That's an obvious conflict-of-interest surface for a benchmark that ranks models: the operator grades the homework. The cheapest credible fix is independent replication — third parties re-running published configurations on their own API keys and submitting the results through the same validation pipeline everyone else uses.
This is an open call for exactly that. Human contributors, AI agents, and AI-driven pipelines are all welcome — submissions are judged entirely by their artifacts, which the pipeline verifies by recomputation (see SUBMISSIONS.md), not by who ran them.
What a validation run is
Each issue labeled validation-run pins one published leaderboard entry: exact dataset, provider/model, sample counts, template/question-set hashes, and the expected score with its 95% bootstrap confidence interval. You:
- Reproduce the run with the pinned configuration using
synthbench run (you pay your own API costs — that independence is the point; each issue carries a cost estimate, typically under $5).
- Submit the result through the normal leaderboard pipeline (web upload or
synthbench submit — see SUBMISSIONS.md). Schema v2 submissions carry raw model responses and a reproducibility block; the server recomputes every metric from your per-question distributions, so nobody has to take your word for anything.
- Comment on the issue using the report template below.
Current open targets
Pick one, comment that you're taking it, and follow its steps:
Multiple independent replications of the same target are welcome — that's the point.
What counts as a successful replication
- Replicated: your recomputed
sps falls inside the reference run's 95% CI (given in each issue), or your CI and the reference CI overlap.
- Divergent: outside that band. This is still a valuable contribution — post it. Providers silently update models; a genuine divergence is a finding about benchmark stability, not a failed submission. We'll investigate rather than close.
Runs use provider-default temperature and no fixed seed (matching the reference runs), so exact score equality is not expected and not required.
Report template (paste as an issue comment)
**Validation report**
- Leaderboard submission ID / link:
- Recomputed sps: X.XXX (95% CI [X.XXX, X.XXX])
- Reference sps: X.XXX (95% CI [X.XXX, X.XXX]) → REPLICATED / DIVERGENT
- question_set_hash matches reference: yes/no
- prompt_template_hash matches reference: yes/no
- parse_failure_rate: X.XXX
- synthbench version (`synthbench --version`):
- Anything unusual (provider errors, retries, rate limits):
Notes for AI-agent contributors
- Each
validation-run issue contains a machine-readable spec block; treat it as authoritative over prose.
Getting submission access (AI agents)
Submitting to the leaderboard requires an identity — that's deliberate. A validation run's value comes from accountable independence, so there is intentionally no unattended account-creation path. Two routes work for agents:
- Operator-minted API key (recommended). Your human operator signs in at synthbench.org/account once, mints a write-only submit-scope key, and provides it to you as
SYNTHBENCH_API_KEY. This is a one-time human step by design; after it, synthbench submit (or synthbench run --submit) is fully scriptable. Keys are rate-limited and auditable per SUBMISSIONS.md.
- GitHub PR path (no synthbench.org account at all). Fork the repo, drop your validated result JSON into
leaderboard-results/, open a PR — identity and rate-limiting anchor on your GitHub account instead. The same validators run in CI. For gated datasets, run scripts/strip-gated-distributions.py first (the GSS issues need no stripping).
If neither is available to you, complete everything up to submission, attach your validated result JSON and validation output to your issue comment, and say so — a maintainer can shepherd it. An unattributed result carries less evidentiary weight, so prefer routes 1–2.
- The web/CLI submission path requires no GitHub CI — prefer it. If you instead open a PR into
leaderboard-results/, note that first-time-contributor workflow runs are held for maintainer approval; this is normal, not a rejection.
- Do not hand-edit result JSON. The validator recomputes all metrics and rejects internally inconsistent files (see "Common failure modes" in SUBMISSIONS.md).
Why
Every entry on the synthbench.org leaderboard was produced or triaged by the maintainer team. That's an obvious conflict-of-interest surface for a benchmark that ranks models: the operator grades the homework. The cheapest credible fix is independent replication — third parties re-running published configurations on their own API keys and submitting the results through the same validation pipeline everyone else uses.
This is an open call for exactly that. Human contributors, AI agents, and AI-driven pipelines are all welcome — submissions are judged entirely by their artifacts, which the pipeline verifies by recomputation (see SUBMISSIONS.md), not by who ran them.
What a validation run is
Each issue labeled
validation-runpins one published leaderboard entry: exact dataset, provider/model, sample counts, template/question-set hashes, and the expected score with its 95% bootstrap confidence interval. You:synthbench run(you pay your own API costs — that independence is the point; each issue carries a cost estimate, typically under $5).synthbench submit— see SUBMISSIONS.md). Schema v2 submissions carry raw model responses and a reproducibility block; the server recomputes every metric from your per-question distributions, so nobody has to take your word for anything.Current open targets
Pick one, comment that you're taking it, and follow its steps:
Multiple independent replications of the same target are welcome — that's the point.
What counts as a successful replication
spsfalls inside the reference run's 95% CI (given in each issue), or your CI and the reference CI overlap.Runs use provider-default temperature and no fixed seed (matching the reference runs), so exact score equality is not expected and not required.
Report template (paste as an issue comment)
Notes for AI-agent contributors
validation-runissue contains a machine-readablespecblock; treat it as authoritative over prose.Getting submission access (AI agents)
Submitting to the leaderboard requires an identity — that's deliberate. A validation run's value comes from accountable independence, so there is intentionally no unattended account-creation path. Two routes work for agents:
SYNTHBENCH_API_KEY. This is a one-time human step by design; after it,synthbench submit(orsynthbench run --submit) is fully scriptable. Keys are rate-limited and auditable per SUBMISSIONS.md.leaderboard-results/, open a PR — identity and rate-limiting anchor on your GitHub account instead. The same validators run in CI. For gated datasets, runscripts/strip-gated-distributions.pyfirst (the GSS issues need no stripping).If neither is available to you, complete everything up to submission, attach your validated result JSON and validation output to your issue comment, and say so — a maintainer can shepherd it. An unattributed result carries less evidentiary weight, so prefer routes 1–2.
leaderboard-results/, note that first-time-contributor workflow runs are held for maintainer approval; this is normal, not a rejection.