Bug Description
Benchmark eval/compare scripts write run-scoped artifacts to fixed, shared paths under /tmp. Any two eval processes running concurrently on the same host (e.g. capability 2 and capability 3 of M3 in parallel on different REGISTRY_PORTs — a supported setup since #17's port parameterization) write to the same files, so their console logs interleave, one run's FINAL SUMMARY can be echoed (and bundled) by the other run, and reproducibility bundles capture a mixture of both runs' output.
M3 paths (all fixed, no run scoping):
| Path |
Written by |
Read by |
/tmp/m3_console.log |
benchmarks/m3/eval.sh:389 (exec > >(tee "$CONSOLE_LOG")) |
bundle assembly in eval.sh; compare.sh:625 log staging |
/tmp/m3_summary.txt |
benchmarks/m3/eval_m3.py:382 (M3_SUMMARY_FILE) |
eval.sh:554 FINAL SUMMARY tail block (and rm -f at eval.sh:395 — one run deletes the other's pending summary) |
/tmp/m3_registry.log |
eval.sh:425 (outer-registry path), run_with_container.sh:141 |
bundle --log-files fallback (eval.sh:218,300), compare.sh:632 |
/tmp/m3_groundedness_arm.txt |
run_groundedness_ab.sh:72 |
same script |
The same pattern exists in the other benchmarks (benchmarks/appworld/{eval,compare}.sh, benchmarks/bpo/{eval,compare}.sh, benchmarks/oak_health_insurance/{eval,compare}.sh — fixed /tmp/<bench>_console.log etc.).
Note the per-run experiment workspaces from #11/#105 do not solve this: the workspace finalize step copies from these shared /tmp paths, so a parallel run's interleaved/overwritten content is what gets preserved.
Steps to Reproduce
- Start two M3 predictions-only runs in parallel on one host, one per capability, with distinct registry/demo ports:
# terminal 1
bash benchmarks/m3/eval.sh --m3-data vendor/vakra/data/test/capability_2_dashboard_apis \
--no-ground-truth --capability m3_task_2 --experiment cap2
# terminal 2
REGISTRY_PORT=8011 DYNACONF_SERVER_PORTS__DEMO=7861 \
bash benchmarks/m3/eval.sh --m3-data vendor/vakra/data/test/capability_3_multihop_reasoning \
--no-ground-truth --capability m3_task_3 --experiment cap3
- Inspect
/tmp/m3_console.log while both run, and each experiment workspace's staged console log after they finish.
Expected Behavior
Each run's console log, summary file, and registry log are private to that run; bundles/workspaces contain only their own run's output. Concurrent runs on the same host don't interact through /tmp.
Actual Behavior
- Both processes
tee stdout into the same /tmp/m3_console.log; the interleaved file is what both bundles stage.
eval.sh deletes /tmp/m3_summary.txt at startup, so run B's startup can delete run A's not-yet-echoed summary; whichever summary lands last is echoed/bundled by both.
- Registry log fallback paths mix both runs' output.
Environment
- OS: macOS 15 (Darwin 25.5.0) / Linux VMs
- Branch:
main (also present on all current feature branches)
Additional Context
Suggested fix shape (mirrors the pre-set-env-wins convention from #17): each script resolves its run-scoped paths from an overridable env var, defaulting to a fresh mktemp -d per run, e.g. RUN_TMP_DIR="${M3_RUN_TMP_DIR:-$(mktemp -d "${TMPDIR:-/tmp}/m3_run_XXXXXX")}", with CONSOLE_LOG, summary, and registry log inside it; eval_m3.py reads M3_SUMMARY_FILE from the environment; compare.sh exports the dir before invoking eval.sh and stages from it instead of hardcoded /tmp paths.
Related but out of scope here (repo-relative, not /tmp, same concurrent-run hazard within a single checkout): benchmarks/m3/registry_server.log, find_latest_trajectory over a shared logging/trajectory_data/, and the shared results/_vakra/prediction/<domain>.json (domain-name collisions across capabilities — separate issue).
Bug Description
Benchmark eval/compare scripts write run-scoped artifacts to fixed, shared paths under
/tmp. Any two eval processes running concurrently on the same host (e.g. capability 2 and capability 3 of M3 in parallel on differentREGISTRY_PORTs — a supported setup since #17's port parameterization) write to the same files, so their console logs interleave, one run's FINAL SUMMARY can be echoed (and bundled) by the other run, and reproducibility bundles capture a mixture of both runs' output.M3 paths (all fixed, no run scoping):
/tmp/m3_console.logbenchmarks/m3/eval.sh:389(exec > >(tee "$CONSOLE_LOG"))eval.sh;compare.sh:625log staging/tmp/m3_summary.txtbenchmarks/m3/eval_m3.py:382(M3_SUMMARY_FILE)eval.sh:554FINAL SUMMARY tail block (andrm -fateval.sh:395— one run deletes the other's pending summary)/tmp/m3_registry.logeval.sh:425(outer-registry path),run_with_container.sh:141--log-filesfallback (eval.sh:218,300),compare.sh:632/tmp/m3_groundedness_arm.txtrun_groundedness_ab.sh:72The same pattern exists in the other benchmarks (
benchmarks/appworld/{eval,compare}.sh,benchmarks/bpo/{eval,compare}.sh,benchmarks/oak_health_insurance/{eval,compare}.sh— fixed/tmp/<bench>_console.logetc.).Note the per-run experiment workspaces from #11/#105 do not solve this: the workspace finalize step copies from these shared /tmp paths, so a parallel run's interleaved/overwritten content is what gets preserved.
Steps to Reproduce
/tmp/m3_console.logwhile both run, and each experiment workspace's staged console log after they finish.Expected Behavior
Each run's console log, summary file, and registry log are private to that run; bundles/workspaces contain only their own run's output. Concurrent runs on the same host don't interact through
/tmp.Actual Behavior
teestdout into the same/tmp/m3_console.log; the interleaved file is what both bundles stage.eval.shdeletes/tmp/m3_summary.txtat startup, so run B's startup can delete run A's not-yet-echoed summary; whichever summary lands last is echoed/bundled by both.Environment
main(also present on all current feature branches)Additional Context
Suggested fix shape (mirrors the pre-set-env-wins convention from #17): each script resolves its run-scoped paths from an overridable env var, defaulting to a fresh
mktemp -dper run, e.g.RUN_TMP_DIR="${M3_RUN_TMP_DIR:-$(mktemp -d "${TMPDIR:-/tmp}/m3_run_XXXXXX")}", withCONSOLE_LOG, summary, and registry log inside it;eval_m3.pyreadsM3_SUMMARY_FILEfrom the environment;compare.shexports the dir before invokingeval.shand stages from it instead of hardcoded/tmppaths.Related but out of scope here (repo-relative, not /tmp, same concurrent-run hazard within a single checkout):
benchmarks/m3/registry_server.log,find_latest_trajectoryover a sharedlogging/trajectory_data/, and the sharedresults/_vakra/prediction/<domain>.json(domain-name collisions across capabilities — separate issue).