Skip to content

LiveMath scores imply 125 test items, but the released split has 124 #157

Description

@KhanCold

Hi SkillOpt team,

I found a possible mismatch between the LiveMath split and the Qwen results in Table 1.

The released manifest contains 35/18/124 train/val/test items. However, several reported scores exactly match a denominator of 125:

  • Qwen3.5-4B: No skill 22.4 = 28/125, Trace2Skill 27.2 = 34/125, SkillOpt 52.0 = 65/125
  • Qwen3.6-35B-A3B: No skill 31.2 = 39/125, Trace2Skill 29.6 = 37/125, SkillOpt 41.6 = 52/125

With 124 binary exact-match examples, neither 52.0% nor 41.6% is attainable after rounding.

For 177 total items, a 2:1:7 split can be either:

  • 35/17/125 if train and val are floored and the remainder goes to test;
  • 35/18/124 with the currently released largest-remainder split.

Could you confirm whether Table 1 used the former 35/17/125 split, and release the original LiveMath manifest or per-item results?

Related: #108.

Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions