Skip to content

[Bug]: Hardcoded shared /tmp paths collide between concurrent benchmark runs (console log, summary, registry log) #115

Description

@haroldship

Bug Description

Benchmark eval/compare scripts write run-scoped artifacts to fixed, shared paths under /tmp. Any two eval processes running concurrently on the same host (e.g. capability 2 and capability 3 of M3 in parallel on different REGISTRY_PORTs — a supported setup since #17's port parameterization) write to the same files, so their console logs interleave, one run's FINAL SUMMARY can be echoed (and bundled) by the other run, and reproducibility bundles capture a mixture of both runs' output.

M3 paths (all fixed, no run scoping):

Path Written by Read by
/tmp/m3_console.log benchmarks/m3/eval.sh:389 (exec > >(tee "$CONSOLE_LOG")) bundle assembly in eval.sh; compare.sh:625 log staging
/tmp/m3_summary.txt benchmarks/m3/eval_m3.py:382 (M3_SUMMARY_FILE) eval.sh:554 FINAL SUMMARY tail block (and rm -f at eval.sh:395 — one run deletes the other's pending summary)
/tmp/m3_registry.log eval.sh:425 (outer-registry path), run_with_container.sh:141 bundle --log-files fallback (eval.sh:218,300), compare.sh:632
/tmp/m3_groundedness_arm.txt run_groundedness_ab.sh:72 same script

The same pattern exists in the other benchmarks (benchmarks/appworld/{eval,compare}.sh, benchmarks/bpo/{eval,compare}.sh, benchmarks/oak_health_insurance/{eval,compare}.sh — fixed /tmp/<bench>_console.log etc.).

Note the per-run experiment workspaces from #11/#105 do not solve this: the workspace finalize step copies from these shared /tmp paths, so a parallel run's interleaved/overwritten content is what gets preserved.

Steps to Reproduce

  1. Start two M3 predictions-only runs in parallel on one host, one per capability, with distinct registry/demo ports:
    # terminal 1
    bash benchmarks/m3/eval.sh --m3-data vendor/vakra/data/test/capability_2_dashboard_apis \
      --no-ground-truth --capability m3_task_2 --experiment cap2
    # terminal 2
    REGISTRY_PORT=8011 DYNACONF_SERVER_PORTS__DEMO=7861 \
    bash benchmarks/m3/eval.sh --m3-data vendor/vakra/data/test/capability_3_multihop_reasoning \
      --no-ground-truth --capability m3_task_3 --experiment cap3
  2. Inspect /tmp/m3_console.log while both run, and each experiment workspace's staged console log after they finish.

Expected Behavior

Each run's console log, summary file, and registry log are private to that run; bundles/workspaces contain only their own run's output. Concurrent runs on the same host don't interact through /tmp.

Actual Behavior

  • Both processes tee stdout into the same /tmp/m3_console.log; the interleaved file is what both bundles stage.
  • eval.sh deletes /tmp/m3_summary.txt at startup, so run B's startup can delete run A's not-yet-echoed summary; whichever summary lands last is echoed/bundled by both.
  • Registry log fallback paths mix both runs' output.

Environment

  • OS: macOS 15 (Darwin 25.5.0) / Linux VMs
  • Branch: main (also present on all current feature branches)

Additional Context

Suggested fix shape (mirrors the pre-set-env-wins convention from #17): each script resolves its run-scoped paths from an overridable env var, defaulting to a fresh mktemp -d per run, e.g. RUN_TMP_DIR="${M3_RUN_TMP_DIR:-$(mktemp -d "${TMPDIR:-/tmp}/m3_run_XXXXXX")}", with CONSOLE_LOG, summary, and registry log inside it; eval_m3.py reads M3_SUMMARY_FILE from the environment; compare.sh exports the dir before invoking eval.sh and stages from it instead of hardcoded /tmp paths.

Related but out of scope here (repo-relative, not /tmp, same concurrent-run hazard within a single checkout): benchmarks/m3/registry_server.log, find_latest_trajectory over a shared logging/trajectory_data/, and the shared results/_vakra/prediction/<domain>.json (domain-name collisions across capabilities — separate issue).

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status
In progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions