CrabNet matbench data, possible mismatch between submission notebook and results

[sgbaird]

@MahamadSalah74 I ran the matbench notebook posted on the matbench GitHub page (materialsproject/matbench#23), and got somewhat higher MAEs for a few repeat runs (see below) compared to what was reported.

0.3683
0.3661
0.3658

The reported matbench result (0.3463) seems a bit more in line with matbench_crabnet.py which uses 300 epochs and the full train/val dataset for training. It's not a huge difference, but I'm trying to figure out if/what the discrepancy is.

Maybe I'm missing something basic. Could you comment on this?

[sgbaird]

Also, the CrabNet paper reports an even lower error for the matbench_expt_gap task (0.338 eV) than what's shown in the matbench submission (0.3463 eV). Maybe it is just that I didn't run it for enough epochs. Ran it for 100 epochs and got 0.3485 eV.

[sgbaird]

I think I was using a fresh clone of CrabNet, but this would probably need to be verified.