Welcome — and how to reproduce the benchmarks yourself

[Cyberfilo]

Thanks for stopping by PromptQuery — a CLI that turns plain English into read-only Postgres SQL, built for real schemas with hundreds of tables.

The thing I care about most is that the numbers are honest and reproducible:

• Accuracy (does the generated SQL reference every table the question needs, and invent none?) on two real schemas: 100% on Odoo 18 (675 tables) and 94% on EMBL-EBI RNAcentral (216 tables). The receipts — including the failures — are committed in eval/.
• Token cost: ~12–17× smaller prompts than stuffing the whole schema in, measured with tiktoken via eval/token_bench.py.

Reproduce them yourself:

python -m eval.parsing_bench --fixture eval/fixtures/odoo.schema.json --questions eval.questions.odoo --model gpt-4o --selector-model gpt-4o-mini
python -m eval.token_bench  --fixture eval/fixtures/odoo.schema.json --questions eval.questions.odoo

If you find a question where it picks the wrong tables, I genuinely want to see it — drop it here or open an issue. Design bets (determinism, the two read-only safety layers) are in ARCHITECTURE.md, and good first issues are labeled.

— Filippo