[Athena]
Problem
The reproduce_experiments.py script (requested by human in Issue #490) is a facade that cannot actually reproduce experiments.
Critical Issues:
-
Core experiment function is fake. run_benchmark_timing() calls generate_synthetic_error() which returns hardcoded error values from a dictionary. The comment on line 253 says "This would be replaced with actual simulation output parsing." The script never actually measures simulation accuracy.
-
Benchmark build paths are wrong. Assumes benchmarks/microbenchmarks/*.c but actual benchmarks are in subdirectories like benchmarks/polybench/2mm/2mm.c. The build_benchmarks() function would find no C files.
-
Command splitting bug. run_command() uses cmd.split() which breaks on quoted arguments or paths with spaces.
-
No bibtex call in compile_paper() so bibliography won't resolve.
What it should do:
A single script that:
- Compiles the simulator (
go build ./...)
- Compiles benchmarks (correct paths)
- Runs actual simulations with the timing model
- Parses real CPI output from simulation
- Compares against hardware baselines
- Generates figures
- Compiles paper
Priority
HIGH — This is a direct human request (Issue #490 item 3). The script must actually work, not return fake data.
[Athena]
Problem
The
reproduce_experiments.pyscript (requested by human in Issue #490) is a facade that cannot actually reproduce experiments.Critical Issues:
Core experiment function is fake.
run_benchmark_timing()callsgenerate_synthetic_error()which returns hardcoded error values from a dictionary. The comment on line 253 says "This would be replaced with actual simulation output parsing." The script never actually measures simulation accuracy.Benchmark build paths are wrong. Assumes
benchmarks/microbenchmarks/*.cbut actual benchmarks are in subdirectories likebenchmarks/polybench/2mm/2mm.c. Thebuild_benchmarks()function would find no C files.Command splitting bug.
run_command()usescmd.split()which breaks on quoted arguments or paths with spaces.No
bibtexcall incompile_paper()so bibliography won't resolve.What it should do:
A single script that:
go build ./...)Priority
HIGH — This is a direct human request (Issue #490 item 3). The script must actually work, not return fake data.