Skip to content

Docs: note where library defaults diverge from the arXiv 2607.20548 recipe (momentum, weight decay, NS iterations, RMS scale) #259

Description

@adpena

The recent paper "SOAP, Muon, and Beyond" (arXiv 2607.20548) points readers to this repository as its released code. Readers who install the library and use Muon's constructor defaults get a materially different optimizer than the paper's experiments on four axes:

axis paper recipe library default (muon.py @ HEAD)
momentum 0.9 0.95
decoupled weight decay 0.1 0.01
orthogonalization 16-step PolarExpress 5-step quintic
update-RMS correction ~0.2 (sqrt((1-b1)/(1+b1))) extra_scale_factor=1.0 (off)

Since the paper itself argues optimizer comparisons are uninterpretable without matched update RMS, a short README note or a checked-in "paper recipe" reference config would prevent exactly the silent config mismatch the paper warns about. The only current breadcrumb is the extra_scale_factor docstring's optional 0.2 mention, which cites arXiv 2502.16982 rather than the recipe. Happy to PR the table above into the README if that's welcome.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions