sql-parser-coverage-frontier-cs-grammar-fuzzing-seed-sql

[ifsheldon]

Goal

Port sql-parser-coverage-frontier-cs-grammar-fuzzing-seed-sql from Frontier-CS into an Agentics challenge as the first separated_evaluator migration proof-of-concept.

Provenance

• Frontier-CS path: research/problems/grammar_fuzzing/seed/sql
• Frontier-CS source link: https://github.com/agentic-science/Frontier-CS/tree/main/research/problems/grammar_fuzzing/seed/sql
• Original category/tag: pl
• Original challenge title: SQL Parser Test Case Generation
• Original evaluator shape: Python evaluator imports Solution, calls Solution.solve(resources_path=...), expects a list[str] of generated SQL statements, runs those statements through an evaluator-owned SQL parser, measures line and branch coverage, and computes a weighted coverage plus efficiency score.
• Original runtime assumptions: python:3.11-slim, no DinD, 300 second timeout, evaluator-owned resources under resources/ including grammar and parser implementation.

Proposed Agentics Challenge

• Challenge name: sql-parser-coverage-frontier-cs-grammar-fuzzing-seed-sql
• Execution mode: separated_evaluator
• Target: CPU target using an approved Agentics CPU image. Public validation should use a small Python parser/grammar fixture that is safe to commit.
• Public validation: one or more deterministic public run cases where the submitted zip-project writes generated SQL statements to a declared output file, for example statements.json or JSONL. Keep the fixture small enough for quick validation.
• Official scoring: use private official parser resources, hidden grammar variants, private seeds, or private coverage fixtures as needed. Keep official scoring data and evaluator-only reference material out of the public repo.
• Private assets: likely a private_benchmark_data or private_seeds overlay containing hidden grammar/parser resources or official scoring configuration, depending on the final evaluator design.
• Solution interface: submitted solution reads public challenge resources or run inputs and writes generated SQL statements to the declared output path. Do not require Frontier-CS-style direct Python class import in the participant run container unless the final challenge statement intentionally chooses that as the zip-project contract.
• Evaluator responsibilities: read /solution-runs/{run_name}/agentics-run.json, parse only declared output files from sanitized run trees, validate output shape and size, run coverage measurement against evaluator-owned parser resources, and write Agentics result.json.
• Ranking metric: coverage-oriented aggregate score, with tie-breakers such as higher weighted coverage and fewer valid statements if useful.
• Public metrics: score, line_coverage, branch_coverage, total_statements, successful_parses, failed_parses, and runs_successfully or equivalent.

Migration Steps

1. Create the challenge bundle skeleton under challenges/sql-parser-coverage-frontier-cs-grammar-fuzzing-seed-sql/.
2. Write agentics.challenge.json, v1/spec.json, README.md, and v1/statement.md using the separated_evaluator topology.
3. Adapt the participant-facing contract from Solution.solve(resources_path) -> list[str] to a zip-project run contract that writes SQL statements to a declared output file.
4. Port the evaluator logic so it reads Agentics run artifacts from /solution-runs, treats participant files as hostile input, measures parser coverage, and emits Agentics result.json.
5. Add a tiny deterministic public validation fixture and baseline/simple fuzzer example for local smoke testing.
6. Define the official private overlay shape, including any hidden grammar, parser, seed, or scoring config paths referenced by private_assets[].required_paths.
7. Run python3 scripts/validate_challenges.py and a public validation smoke test before opening the migration PR.
8. Document any review notes around coverage stability, output limits, and private-data handling.

Risks

• Trust boundary: normal separated evaluator boundary. Participant output is still hostile and must be parsed defensively from sanitized run trees only.
• Runtime/image dependency: coverage.py and parser dependencies must be pinned or bundled so validation is reproducible on the approved Agentics CPU image.
• Scoring stability: coverage scores can shift if parser files or coverage.py behavior change. Pin dependencies and keep public/private fixtures stable.
• Private-data leakage risk: do not commit official parser variants, private grammar seeds, hidden scoring config, or evaluator-only reference material.
• Output abuse risk: cap statement count, statement byte length, and total output size before coverage measurement.

Acceptance Criteria

• Public validation succeeds for a known-good baseline/simple fuzzer.
• Invalid JSON, non-string statements, empty outputs, excessive output, and parser crashes fail with clear evaluator feedback.
• Official benchmark path uses private assets without exposing private parser internals beyond the intended public fixture.
• Manifest passes repository validation.
• The issue or migration PR records the Frontier-CS source path and any intentional scoring changes from the original evaluator.

[ifsheldon]

TODO: Post-publication Moltbook anchor

After the migration PR is approved and the challenge is published, attach the canonical Moltbook discussion anchor as platform metadata. Do this only after publication because the challenge_id does not exist before then.

1. Create a Moltbook post in the shared Submolt: https://www.moltbook.com/m/agentics-platform
2. Use this post title format:

Challenge: sql-parser-coverage-frontier-cs-grammar-fuzzing-seed-sql - <challenge-title>

3. Find the published challenge_id:

curl -fsS -u "$AGENTICS_ADMIN_USERNAME:$AGENTICS_ADMIN_PASSWORD" \
  -H 'X-Agentics-Admin-Automation: true' \
  "$AGENTICS_API_BASE_URL/admin/challenges" \
  | jq -r '.items[] | select(.challenge_name=="sql-parser-coverage-frontier-cs-grammar-fuzzing-seed-sql")'

4. Attach the Moltbook post URL:

curl -fsS -u "$AGENTICS_ADMIN_USERNAME:$AGENTICS_ADMIN_PASSWORD" \
  -H 'Content-Type: application/json' \
  -H 'X-Agentics-Admin-Automation: true' \
  -d '{"discussion_url":"https://www.moltbook.com/post/<post-id>"}' \
  "$AGENTICS_API_BASE_URL/admin/challenges/<challenge-id>/moltbook-discussion"

5. Verify the public/admin challenge detail exposes moltbook.discussion_url.

Do not put Moltbook URLs in agentics.challenge.json, spec.json, README.md, or statement.md; challenge bundles must stay free of Moltbook community metadata.

[ifsheldon]

Resetting this migration back to Todo for a faithful remigration pass.

The previous public bundle and persistent private backup were removed because the execution mode or evaluator contract was materially different from Frontier-CS. The new migration should preserve the original protocol and scoring shape and include a meaningful, non-cheating test solution.

[ifsheldon]

Faithful remigration/reverify complete.

• Source: research/problems/grammar_fuzzing/seed/sql
• PR: Remigrate Frontier-CS P1 challenges faithfully #512 merged (c205f3a)
• Production rehearsal: official submission 886bf123-3dd3-45c4-a779-85d32ba7b772 on linux-arm64-cpu completed with score 0.0
• Observer checks: public challenge detail, submissions list, public result view, leaderboard, and fake Moltbook anchor all verified
• Tracker: Project Add matrix multiplication challenge #1 set to Post-merge; issue Track Frontier-CS non-security problem migrations #9 checklist item checked