Add latch-eval CLI for running eval JSONs locally#39
Open
chriswzou wants to merge 8 commits into
Open
Conversation
Adds a `latch-eval run` command so a single eval JSON can be run end-to-end (download data -> run an agent in a Docker sandbox -> grade) without writing any Python — the fast local draft -> run -> grade loop for eval authoring. CLI is split across cli/app.py (parser + main), cli/run.py (the run subcommand + result serialization), cli/harnesses.py (harness dispatch + provider hints), and cli/preflight.py (Docker / API-key / Latch-token checks); cli/__init__.py is a thin re-export so the latch_eval_tools.cli:main entry point is unchanged. Also: - EvalRunner(data_node_override=...) to point a run at alternate data. - Local-path / file:// support in the downloader plus a `--data` flag, so runs can use data already on disk and skip the Latch download entirely; hardlink staging falls back to copy across filesystems. - .gitignore: ignore the real .eval_workspace/ and /workspace/ run artifacts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Run artifacts were rooted at get_project_root(), which for an editable install resolves to the latch-eval-tools checkout — so every run's workspace (and any --workspace-name dirs) got written inside the package repo, regardless of where latch-eval was invoked, and the behavior differed by install mode. The CLI now anchors workspaces to ./.latch-eval-runs/<eval_id>/ relative to the invocation directory, and the dataset cache to a shared ~/.cache/latch-eval — robust to any install mode. - EvalRunner gains work_root and cache_dir; setup_workspace gains root_dir and get_cache_dir/manifest/download_* thread an explicit cache_dir. All default to None, preserving the project-root layout for library callers (scbench/spatialbench). - CLI: replace --workspace-name/--cache-name with --output-dir (default ./.latch-eval-runs) and --cache-dir (default ~/.cache/latch-eval). - README + .gitignore updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…dent of the active env Lead the install instructions with uv tool / pipx (isolated, global PATH, not tied to an active conda/venv or to pip), including local-checkout and git-branch installs for the pre-publish CLI. Keep pip-into-env as a secondary option with a note that the command is only on PATH while that env is active. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The latch-eval console script (via cli/__init__.py) and library importability cover the use cases; the python -m latch_eval_tools.cli invocation isn't needed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Use the textbook entry-point form (module:callable at the definition site) instead of re-exporting main through the package __init__. cli/__init__.py is now just a docstring. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collapse the run/harnesses/preflight modules into a single cli/runner.py (run_command, harness dispatch, and preflight together); app.py keeps the parser and main. Fix the harness import to pull the run_*_task functions from latch_eval_tools.harness (their public location) rather than harness._cli_runner. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Generic version of the /run-eval Claude Code skill (drop into .claude/skills/run-eval/SKILL.md), kept in sync with the current CLI: --output-dir/--cache-dir, ./.latch-eval-runs/ workspaces, ~/.cache/latch-eval, and uv tool install on PATH. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CLI now discovers a .env from the current directory upward and loads it (without overriding the real environment) so provider keys like ANTHROPIC_API_KEY are picked up without sourcing/exporting first; --env-file points at a specific file. Uses find_dotenv(usecwd=True) so discovery is relative to where the command is invoked, not the package location. Adds python-dotenv as a dependency. Also gitignore .env (and .env.* except .env.example), and document the behavior in the README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AnirudhNarsipur
requested changes
Jun 12, 2026
| "mini-swe-agent~=2.2", | ||
| "litellm~=1.87.0", | ||
| "jinja2~=3.1", | ||
| "python-dotenv>=1.0.0", |
Contributor
There was a problem hiding this comment.
You need to regenerate the lock file as well. Also bump project version number.
| self.cache_name = cache_name | ||
| self.workspace_name = workspace_name | ||
| self.benchmark_name = benchmark_name | ||
| self.data_node_override = data_node_override |
Contributor
There was a problem hiding this comment.
Overriding data nodes is fine but this should be tracked in the metadata generated in the results - see bottom of this function
| staged in place without any download. | ||
| """ | ||
| if uri.startswith("file://"): | ||
| return False |
Contributor
There was a problem hiding this comment.
nit: use urllib parse or smth?
| def main(argv=None): | ||
| parser = build_parser() | ||
| args = parser.parse_args(argv) | ||
| _load_dotenv(args.env_file) |
Contributor
There was a problem hiding this comment.
nit: should resolve env file path?
| cache_dir = get_cache_dir(cache_name) | ||
| manifest = get_cache_manifest(cache_name) | ||
| if not _is_remote_uri(uri): | ||
| raw = uri[len("file://"):] if uri.startswith("file://") else uri |
Contributor
There was a problem hiding this comment.
nit: just parse the uri? this is quite weird. Also don't think this works for parsing literal uris file://my%20data.csv but idc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
latch-eval runcommand so a single eval JSON can be run end-to-end — download itsdata_node, run an agent in a Docker sandbox, captureeval_answer.json, and grade it — without writing any Python. This is the fast local draft → run → grade loop for eval authoring; previously the only way to execute an eval was hand-writingEvalRunner(...).run(agent_function=...).This is largely intended for use by Codex/Claude Code (the args are so annoying to configure otherwise), and to help with that there's a handy little skill template included in the
clidirectory.latch-eval run -e evals/QC01.json --harness claudecode latch-eval run -e evals/de03.json --harness minisweagent --model anthropic/claude-sonnet-4-6 latch-eval run -e evals/de03.json --harness claudecode --data ./local/data.tsv.gz # skip Latch download