Follow-up to the Phase 0 turn-trace baseline (#700). The baseline lands with a per-oracle-tier pass/fail split so the report cannot be misread as a blanket correctness verdict:
- Correctness (18 tasks: edit + fix) — positive oracles (substring grep, scoped
go test). The only pass rate that can move with model quality.
- Build-only (6 tasks: refactor) —
go build ./..., reported as buildPassRate, excluded from correctnessPassRate.
- Latency-only (24 tasks: nav/longproc/longctx/parallel) — no oracle; counted in
latencyOnlyTasks, excluded from every pass rate.
The tier split keeps the report honest, so this hardening is not a blocker — but the oracles are intentionally lightweight for Phase 0 and should be strengthened before a baseline report is treated as a correctness signal.
Work
- Edit grep oracles are substring checks. An add-field oracle (
grep -R Label .) only proves the string appears, not that it landed on the right struct. Add go vet/go build plus a structural assertion (e.g. the field on the named struct) for add-field/add-method tasks. Rename oracles already assert the old name is gone.
- Refactor build oracle is non-positive.
go build ./... passes a no-op refactor. Add a structural verifier that proves the refactor happened (extracted helper exists, call sites updated), or document refactor as permanently structure-uncorrected.
- 24 read-only tasks have no oracle. For nav, a deterministic verifier is realistic (
diff -q <(expected) <(agent-output) if the harness pipes the model's final text to a known path). longproc/longctx/parallel summarize with no canonical answer — document them as permanently latency-only.
Where
- Manifest:
internal/perfbench/manifests/baseline.json
- Runner / tier logic:
internal/perfbench/turn_bench.go (NewTurnExecRunner, RunTurnBench)
- Tier contract:
internal/perfbench/MANIFEST.md
Follow-up to the Phase 0 turn-trace baseline (#700). The baseline lands with a per-oracle-tier pass/fail split so the report cannot be misread as a blanket correctness verdict:
go test). The only pass rate that can move with model quality.go build ./..., reported asbuildPassRate, excluded fromcorrectnessPassRate.latencyOnlyTasks, excluded from every pass rate.The tier split keeps the report honest, so this hardening is not a blocker — but the oracles are intentionally lightweight for Phase 0 and should be strengthened before a baseline report is treated as a correctness signal.
Work
grep -R Label .) only proves the string appears, not that it landed on the right struct. Addgo vet/go buildplus a structural assertion (e.g. the field on the named struct) for add-field/add-method tasks. Rename oracles already assert the old name is gone.go build ./...passes a no-op refactor. Add a structural verifier that proves the refactor happened (extracted helper exists, call sites updated), or document refactor as permanently structure-uncorrected.diff -q <(expected) <(agent-output)if the harness pipes the model's final text to a known path). longproc/longctx/parallel summarize with no canonical answer — document them as permanently latency-only.Where
internal/perfbench/manifests/baseline.jsoninternal/perfbench/turn_bench.go(NewTurnExecRunner,RunTurnBench)internal/perfbench/MANIFEST.md