Skip to content

Comments

Add README documentation#79

Merged
Ki-Seki merged 5 commits intomainfrom
copilot/add-simple-readme
Feb 10, 2026
Merged

Add README documentation#79
Ki-Seki merged 5 commits intomainfrom
copilot/add-simple-readme

Conversation

Copy link
Contributor

Copilot AI commented Feb 9, 2026

The repository lacked user-facing documentation.

Changes

Added README.md covering:

  • Project description (benchmarking framework for Guided Infilling Models)
  • Installation via pip (regular and dev)
  • Available benchmark types: CV parsing, regex matching, MCQA, perplexity
  • Command examples for running MMLU-Pro, GPQA Diamond, and GIM-SFT benchmarks
  • Development commands: linting, formatting, pre-commit
Original prompt

add a simple readme


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Co-authored-by: Ki-Seki <60967965+Ki-Seki@users.noreply.github.com>
Copilot AI changed the title [WIP] Add a simple README file Add README documentation Feb 9, 2026
Copilot AI requested a review from Ki-Seki February 9, 2026 23:48
@Ki-Seki
Copy link
Member

Ki-Seki commented Feb 9, 2026

  • install with pip is preferred for common users.
  • add command examples
  • remove license sections

@copilot

…ense sections

Co-authored-by: Ki-Seki <60967965+Ki-Seki@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 9, 2026

  • install with pip is preferred for common users.
  • add command examples
  • remove license sections ...

Updated in commit b72eeb9:

  • Installation now uses pip instead of uv
  • Added command examples for MMLU-Pro, GPQA Diamond, and GIM-SFT benchmarks
  • Removed License and Copyright sections

@Ki-Seki Ki-Seki marked this pull request as ready for review February 10, 2026 00:04
Copilot AI review requested due to automatic review settings February 10, 2026 00:04
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new top-level README to provide user-facing documentation for GIMBench (a benchmarking framework for Guided Infilling Models), including installation, benchmark types, example run commands, and basic development workflows.

Changes:

  • Added project overview and high-level feature/benchmark descriptions.
  • Documented installation steps (pip + dev setup) and example benchmark invocation commands.
  • Documented common development commands (lint, lint-fix, pre-commit) via make.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +17 to +21
For development:

```bash
make install-dev
```
Copy link

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The development install instructions rely on make install-dev, which runs uv sync (see Makefile). Consider documenting uv as a prerequisite (or provide an alternative dev install command) so the steps are reproducible for new contributors.

Copilot uses AI. Check for mistakes.
```bash
python -m gimbench.mcqa.gpqa_diamond \
--model_type openai \
--model_name gpt-4 \
Copy link

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GPQA example uses --model_type openai with gpt-4 but does not specify --base_url. The CLI default base URL is http://localhost:8000/v1, so this command would point at a local endpoint rather than the OpenAI API unless clarified. Suggest adding an explicit OpenAI base URL (or noting that openai here means an OpenAI-compatible local server).

Suggested change
--model_name gpt-4 \
--model_name gpt-4 \
--base_url https://api.openai.com/v1 \

Copilot uses AI. Check for mistakes.
@Ki-Seki Ki-Seki merged commit 9049c9d into main Feb 10, 2026
3 checks passed
@Ki-Seki Ki-Seki deleted the copilot/add-simple-readme branch February 10, 2026 00:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants