Hi SkillOpt team,
I found a possible mismatch between the LiveMath split and the Qwen results in Table 1.
The released manifest contains 35/18/124 train/val/test items. However, several reported scores exactly match a denominator of 125:
- Qwen3.5-4B: No skill
22.4 = 28/125, Trace2Skill 27.2 = 34/125, SkillOpt 52.0 = 65/125
- Qwen3.6-35B-A3B: No skill
31.2 = 39/125, Trace2Skill 29.6 = 37/125, SkillOpt 41.6 = 52/125
With 124 binary exact-match examples, neither 52.0% nor 41.6% is attainable after rounding.
For 177 total items, a 2:1:7 split can be either:
35/17/125 if train and val are floored and the remainder goes to test;
35/18/124 with the currently released largest-remainder split.
Could you confirm whether Table 1 used the former 35/17/125 split, and release the original LiveMath manifest or per-item results?
Related: #108.
Thanks!
Hi SkillOpt team,
I found a possible mismatch between the LiveMath split and the Qwen results in Table 1.
The released manifest contains
35/18/124train/val/test items. However, several reported scores exactly match a denominator of 125:22.4 = 28/125, Trace2Skill27.2 = 34/125, SkillOpt52.0 = 65/12531.2 = 39/125, Trace2Skill29.6 = 37/125, SkillOpt41.6 = 52/125With 124 binary exact-match examples, neither
52.0%nor41.6%is attainable after rounding.For 177 total items, a
2:1:7split can be either:35/17/125if train and val are floored and the remainder goes to test;35/18/124with the currently released largest-remainder split.Could you confirm whether Table 1 used the former
35/17/125split, and release the original LiveMath manifest or per-item results?Related: #108.
Thanks!