Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/scripts/verify-ci-checks.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# concurrency group cancels one of the twin runs — park a cancelled or
# failed run next to the successful same-named run, and those superseded
# runs never turn green. Counting all of them blocked SynthBench v0.3.1
# (PR #334) and SynthPanel's first v1.6.0 attempt, forcing manual tags.
# (PR #334) and Althing's first v1.6.0 attempt, forcing manual tags.
# Only the newest run per (app, name) reflects the commit's real status;
# GitHub's own merge box collapses check runs the same way.
#
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ jobs:
- run: ruff check .
- run: ruff format --check .
- name: Guard against hyphenated sibling-project regression (sb-1el9)
# The sibling project's canonical names are `synthpanel` (PyPI package)
# and `SynthPanel` (GitHub repo). The stale pre-rename hyphenated form
# The sibling project's canonical names are `althing` (PyPI package)
# and `Althing` (GitHub repo). The stale pre-rename hyphenated form
# should never reappear. Historical audit notes (post-flip-audit-*.md)
# are allowed to reference the old name while documenting the rename.
# The pattern is assembled at runtime so this step doesn't match itself.
Expand All @@ -46,7 +46,7 @@ jobs:
--exclude="post-flip-audit-*.md" \
. || true)
if [ -n "$matches" ]; then
echo "::error::Hyphenated sibling-project name found — use 'synthpanel' (PyPI) or 'SynthPanel' (repo)."
echo "::error::Hyphenated sibling-project name found — use 'althing' (PyPI) or 'Althing' (repo)."
echo "$matches"
exit 1
fi
Expand Down Expand Up @@ -329,7 +329,7 @@ jobs:
- name: Mint App token
# Pushing with this token (instead of GITHUB_TOKEN) makes the
# baseline commit fire `pull_request: synchronize`, so CI re-runs on
# the new HEAD automatically. Same pattern as SynthPanel's
# the new HEAD automatically. Same pattern as Althing's
# auto-tag.yml release-bot push.
if: steps.vrt_gate.outputs.run == 'true'
id: app_token
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/semver-label.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ name: Semver Label
# (semver:patch / semver:minor / semver:major to bump, semver:skip to opt
# out) BEFORE merge. auto-tag.yml reads the label post-merge to cut the
# release; without this gate a missing label silently skips the release
# behind a green check (the failure mode SynthPanel hit twice — sp-42i,
# behind a green check (the failure mode Althing hit twice — sp-42i,
# sy-73j — before adopting the same pre-merge check).

on:
Expand Down
38 changes: 22 additions & 16 deletions FINDINGS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@

**Date**: 2026-04-12 (session-1 experiments) | **Numbers regenerated from artifacts** — see below

> **Naming note (2026-07):** the `synthpanel` provider referenced throughout
> these findings was renamed to **[althing](https://althing.dev)**. Result
> artifacts and generated tables retain the provider id `synthpanel` they
> were recorded under; `--provider synthpanel` remains a working alias for
> the `althing` provider.

> All quantitative tables in this document sit between
> `<!-- BEGIN GENERATED -->` / `<!-- END GENERATED -->` markers and are
> rendered by `scripts/generate-findings-md.py` from the per-question rows
Expand Down Expand Up @@ -34,9 +40,9 @@ asymmetries in how LLMs represent different demographic groups.

| Dataset | 3-model ensemble SPS | Best single model | Random baseline |
|---------|---------------------|-------------------|-----------------|
| globalopinionqa (n=100) | **0.813** | 0.786 (SynthPanel (GPT-4o-mini), product) | 0.774 (n=100) |
| globalopinionqa (n=100) | **0.813** | 0.786 (Althing (GPT-4o-mini), product) | 0.774 (n=100) |
| opinionsqa (n=684) | **0.877** | 0.829 (Gemini 2.5 Flash, raw) | 0.763 (n=684) |
| subpop (n=200) | **0.858** | 0.821 (SynthPanel (Gemini Flash Lite), product) | 0.757 (n=200) |
| subpop (n=200) | **0.858** | 0.821 (Althing (Gemini Flash Lite), product) | 0.757 (n=200) |
<!-- END GENERATED: headline -->

Note on interpretation: SPS is a composite parity score, not a
Expand Down Expand Up @@ -140,7 +146,7 @@ run vs the identical configuration with schema-forced structured capture
count — the elicitation surface is the only variable.

<!-- BEGIN GENERATED: elicitation -->
**SynthPanel (Haiku 4.5)** (product) on gss — n=75 questions × 30 samples per arm, refusal detector v2; the arms differ only in elicitation mode:
**Althing (Haiku 4.5)** (product) on gss — n=75 questions × 30 samples per arm, refusal detector v2; the arms differ only in elicitation mode:

| Metric | Natural | Structured | Δ (structured − natural) |
|--------|---------|------------|--------------------------|
Expand Down Expand Up @@ -252,9 +258,9 @@ three runs costs roughly 3× a single-model run.
<!-- BEGIN GENERATED: ensemble -->
| Dataset | Best single model | Equal blend | Improvement | Random baseline |
|---------|-------------------|-------------|-------------|-----------------|
| globalopinionqa (100q) | 0.786 (SynthPanel (GPT-4o-mini), product) | **0.813** | **+2.7 pts** | 0.774 (n=100) |
| globalopinionqa (100q) | 0.786 (Althing (GPT-4o-mini), product) | **0.813** | **+2.7 pts** | 0.774 (n=100) |
| opinionsqa (684q) | 0.829 (Gemini 2.5 Flash, raw) | **0.877** | **+4.8 pts** | 0.763 (n=684) |
| subpop (200q) | 0.821 (SynthPanel (Gemini Flash Lite), product) | **0.858** | **+3.7 pts** | 0.757 (n=200) |
| subpop (200q) | 0.821 (Althing (Gemini Flash Lite), product) | **0.858** | **+3.7 pts** | 0.757 (n=200) |

Comparison set: best_single = highest recomputed-SPS non-ensemble, non-baseline leaderboard row (raw or product framework) evaluated on at least as many questions as the ensemble, after per-(model, framework, dataset) dedup; random_baseline = recomputed SPS of the random-baseline run for the same dataset (n in random_baseline_n).
<!-- END GENERATED: ensemble -->
Expand All @@ -277,25 +283,25 @@ artifacts strip `human_distribution` (#308).

| Constituent pair | Pearson r (per-question JSD) | n |
|------------------|------------------------------|---|
| SynthPanel (Haiku 4.5) ↔ SynthPanel (Gemini Flash Lite) | 0.419 | 100 |
| SynthPanel (Haiku 4.5) ↔ SynthPanel (GPT-4o-mini) | 0.308 | 100 |
| SynthPanel (Gemini Flash Lite) ↔ SynthPanel (GPT-4o-mini) | 0.493 | 100 |
| Althing (Haiku 4.5) ↔ Althing (Gemini Flash Lite) | 0.419 | 100 |
| Althing (Haiku 4.5) ↔ Althing (GPT-4o-mini) | 0.308 | 100 |
| Althing (Gemini Flash Lite) ↔ Althing (GPT-4o-mini) | 0.493 | 100 |

**opinionsqa** (n=684 common questions, mean pairwise r = 0.295):

| Constituent pair | Pearson r (per-question JSD) | n |
|------------------|------------------------------|---|
| SynthPanel (Haiku 4.5) ↔ SynthPanel (Gemini Flash Lite) | 0.224 | 684 |
| SynthPanel (Haiku 4.5) ↔ SynthPanel (GPT-4o-mini) | 0.355 | 684 |
| SynthPanel (Gemini Flash Lite) ↔ SynthPanel (GPT-4o-mini) | 0.308 | 684 |
| Althing (Haiku 4.5) ↔ Althing (Gemini Flash Lite) | 0.224 | 684 |
| Althing (Haiku 4.5) ↔ Althing (GPT-4o-mini) | 0.355 | 684 |
| Althing (Gemini Flash Lite) ↔ Althing (GPT-4o-mini) | 0.308 | 684 |

**subpop** (n=200 common questions, mean pairwise r = 0.322):

| Constituent pair | Pearson r (per-question JSD) | n |
|------------------|------------------------------|---|
| SynthPanel (Haiku 4.5) ↔ SynthPanel (Gemini Flash Lite) | 0.282 | 200 |
| SynthPanel (Haiku 4.5) ↔ SynthPanel (GPT-4o-mini) | 0.260 | 200 |
| SynthPanel (Gemini Flash Lite) ↔ SynthPanel (GPT-4o-mini) | 0.425 | 200 |
| Althing (Haiku 4.5) ↔ Althing (Gemini Flash Lite) | 0.282 | 200 |
| Althing (Haiku 4.5) ↔ Althing (GPT-4o-mini) | 0.260 | 200 |
| Althing (Gemini Flash Lite) ↔ Althing (GPT-4o-mini) | 0.425 | 200 |

The constituents' errors are **moderately positively correlated** (pairwise r 0.22–0.49 on per-question JSD; 0.27–0.44 on signed per-option residuals, see asserted constants) — not uncorrelated. The ensemble gain comes from partial, not full, independence of errors; the earlier "uncorrelated errors" framing overstated it.

Expand Down Expand Up @@ -403,8 +409,8 @@ gated files are committed without `human_distribution`).
| Provider | Framework | Template | Dataset | Mean model nonresponse | Mean human nonresponse | Mean abs gap | n |
|----------|-----------|----------|---------|------------------------|------------------------|--------------|---|
| Gemini 2.5 Flash | raw | default | gss | 0.106 | 0.056 | **0.126** | 75 |
| SynthPanel (Haiku 4.5) | product | structured | gss | 0.064 | 0.056 | **0.095** | 75 |
| SynthPanel (Haiku 4.5) | product | default | gss | 0.067 | 0.056 | **0.094** | 75 |
| Althing (Haiku 4.5) | product | structured | gss | 0.064 | 0.056 | **0.095** | 75 |
| Althing (Haiku 4.5) | product | default | gss | 0.067 | 0.056 | **0.094** | 75 |

Comparison set: Per deduped non-baseline run whose committed per-question rows still carry human_distribution (full-tier datasets; gated files are stripped per #308): mean |model explicit-nonresponse mass (DK-style option mass + parsed refusal rate) - human's|, plus the five items with the largest model over-selection.

Expand Down
18 changes: 9 additions & 9 deletions METHODOLOGY.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ For the full technical details, read on.

### 1.1 Analysis of the Two Frameworks

**Proposed (synthpanel datascientist) — 4 sub-scores:**
**Proposed (althing datascientist) — 4 sub-scores:**
| Metric | Measures | Method |
|--------|----------|--------|
| P_dist | Distributional parity | Jensen-Shannon divergence |
Expand Down Expand Up @@ -144,7 +144,7 @@ default distribution. P_cond isolates the *conditioning mechanism itself*.

**Conditioning protocol**: Each provider is tested with the persona spec it
supports. For raw LLMs, we test all three OpinionsQA styles (QA, BIO, PORTRAY)
and take the max. For dedicated providers (synthpanel, Ditto), we use their
and take the max. For dedicated providers (althing, Ditto), we use their
native persona interface.

#### P_sub — Subgroup Consistency
Expand Down Expand Up @@ -269,7 +269,7 @@ class Distribution:
@dataclass
class ProviderMetadata:
"""Provider identification and capabilities."""
name: str # e.g. "synthpanel", "openai-gpt4o"
name: str # e.g. "althing", "openai-gpt4o"
version: str # Semantic version or model identifier
supports_logprobs: bool # Can return token log-probabilities
supports_distribution: bool # Can return Distribution directly
Expand Down Expand Up @@ -329,7 +329,7 @@ class SynthRespondent(Protocol):

| Provider | Adapter | Distribution Method | Notes |
|----------|---------|-------------------|-------|
| **synthpanel** | `SynthPanelAdapter` | Reported (native API returns distribution) | Our reference provider |
| **althing** | `AlthingAdapter` | Reported (native API returns distribution) | Our reference provider |
| **OpenAI (GPT-4o, etc.)** | `OpenAIAdapter` | Logprobs (top-k token logprobs) | `conditioning_style` mapped to system/user prompt |
| **Anthropic (Claude)** | `AnthropicAdapter` | Sampling (no logprobs available) | Requires n_samples >= 30 for stable distributions |
| **Ditto** | `DittoAdapter` | Reported (if API returns distribution) or Sampling | Pending API access to confirm capabilities |
Expand Down Expand Up @@ -372,7 +372,7 @@ class OpenAIAdapter:
CLI invocation:
```bash
synthbench run --provider openai --model gpt-4o --suite core
synthbench run --provider synthpanel --suite full
synthbench run --provider althing --suite full
synthbench run --provider generic-http --config ./my-provider.yaml --suite core
```

Expand Down Expand Up @@ -617,9 +617,9 @@ Every SynthBench run produces a JSON artifact:
"timestamp": "2026-04-10T12:00:00Z",
"suite": "core",
"provider": {
"name": "synthpanel",
"name": "althing",
"version": "2.1.0",
"adapter": "SynthPanelAdapter",
"adapter": "AlthingAdapter",
"config_hash": "sha256:..."
},
"scores": {
Expand Down Expand Up @@ -669,7 +669,7 @@ Every SynthBench run produces a JSON artifact:
================================================================
SYNTHBENCH v1.0 SCORE CARD
================================================================
Provider: synthpanel v2.1.0
Provider: althing v2.1.0
Suite: SynthBench-Core (300 questions)
Date: 2026-04-10
Run ID: run-20260410-a1b2c3
Expand Down Expand Up @@ -720,7 +720,7 @@ For multi-provider comparisons:
```
| Provider | Version | SPS | P_dist | P_rank | P_cond | P_sub | P_refuse | Suite |
|-----------------|---------|------|--------|--------|--------|-------|----------|-------|
| synthpanel | 2.1.0 | 0.74 | 0.81 | 0.77 | 0.68 | 0.72 | 0.71 | core |
| althing | 2.1.0 | 0.74 | 0.81 | 0.77 | 0.68 | 0.72 | 0.71 | core |
| GPT-4o (portray)| 2026-03 | 0.61 | 0.67 | 0.63 | 0.55 | 0.58 | 0.62 | core |
| Claude (portray)| sonnet | 0.59 | 0.65 | 0.61 | 0.52 | 0.60 | 0.58 | core |
| GPT-4o (raw) | 2026-03 | 0.48 | 0.55 | 0.52 | 0.00 | 0.45 | 0.40 | core |
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Open benchmark harness for synthetic survey respondent quality.

**The MLPerf of synthetic UXR.**

SynthBench measures how well synthetic respondent systems (like [synthpanel](https://github.com/DataViking-Tech/SynthPanel), Ditto, Synthetic Users, or raw ChatGPT prompting) reproduce real human survey response patterns against real Pew American Trends Panel and GlobalOpinionQA ground truth — so "it sounds plausible" gets replaced with a measurable similarity score.
SynthBench measures how well synthetic respondent systems (like [althing](https://github.com/DataViking-Tech/Althing), Ditto, Synthetic Users, or raw ChatGPT prompting) reproduce real human survey response patterns against real Pew American Trends Panel and GlobalOpinionQA ground truth — so "it sounds plausible" gets replaced with a measurable similarity score.

## Quick Start

Expand Down Expand Up @@ -203,7 +203,7 @@ The leaderboard JSON carries per-row cost fields and a top-level
runs listed in `config.ensemble_sources`.
- `pricing_snapshot` records the per-model `input_per_1m` / `output_per_1m`
rates used for this publish run, the `snapshot_date` anchor comment, and
the installed `synth_panel_version` that produced the rates.
the installed `althing_version` that produced the rates.

This lets downstream consumers audit which pricing table produced which
`cost_usd` and reconcile against provider-reported billing without guessing
Expand All @@ -220,7 +220,7 @@ both curves side-by-side.

See [`docs/convergence-analysis.md`](docs/convergence-analysis.md) for the
JSON schema, CLI flags, and the `load_convergence_baseline` integration
surface that synthpanel's `--calibrate-against DATASET:QUESTION` flag
surface that althing's `--calibrate-against DATASET:QUESTION` flag
consumes.

## Citation
Expand Down
2 changes: 1 addition & 1 deletion crew/architect/docs/accessibility-guidelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ Color-only encoding fails for ~8% of male users (colorblindness). For small stac
### Badges

- `bg-accent/15 text-accent` — acceptable with the darker accent token (verify per-instance).
- Provider tags (`bg-provider-synthpanel/10 text-provider-synthpanel`) — provider colors are hex literals; check each one against the surfaces they sit on.
- Provider tags (`bg-provider-althing/10 text-provider-althing`) — provider colors are hex literals; check each one against the surfaces they sit on.

---

Expand Down
Loading
Loading