Conversation
Co-authored-by: Ki-Seki <60967965+Ki-Seki@users.noreply.github.com>
|
…ense sections Co-authored-by: Ki-Seki <60967965+Ki-Seki@users.noreply.github.com>
Updated in commit b72eeb9:
|
There was a problem hiding this comment.
Pull request overview
Adds a new top-level README to provide user-facing documentation for GIMBench (a benchmarking framework for Guided Infilling Models), including installation, benchmark types, example run commands, and basic development workflows.
Changes:
- Added project overview and high-level feature/benchmark descriptions.
- Documented installation steps (pip + dev setup) and example benchmark invocation commands.
- Documented common development commands (lint, lint-fix, pre-commit) via
make.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| For development: | ||
|
|
||
| ```bash | ||
| make install-dev | ||
| ``` |
There was a problem hiding this comment.
The development install instructions rely on make install-dev, which runs uv sync (see Makefile). Consider documenting uv as a prerequisite (or provide an alternative dev install command) so the steps are reproducible for new contributors.
| ```bash | ||
| python -m gimbench.mcqa.gpqa_diamond \ | ||
| --model_type openai \ | ||
| --model_name gpt-4 \ |
There was a problem hiding this comment.
The GPQA example uses --model_type openai with gpt-4 but does not specify --base_url. The CLI default base URL is http://localhost:8000/v1, so this command would point at a local endpoint rather than the OpenAI API unless clarified. Suggest adding an explicit OpenAI base URL (or noting that openai here means an OpenAI-compatible local server).
| --model_name gpt-4 \ | |
| --model_name gpt-4 \ | |
| --base_url https://api.openai.com/v1 \ |
The repository lacked user-facing documentation.
Changes
Added
README.mdcovering:pip(regular and dev)Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.