Fix moe layers to transform

[MichalMraz]

Fixes #3016

What

layers_to_transform filtering could treat .experts.<idx> as the layer index for MoE module names, leading to
incorrect selection of expert modules.

Expected behavior

A module like model.layers.1.mlp.experts.0.up_proj should be targeted iff 1 in layers_to_transform
(regardless of expert index).

Changes

• Adjust selection logic to ignore expert indices when applying layers_to_transform
• Add regression test covering MoE module paths

[BenjaminBossan]

Thanks for reporting this issue and providing a fix. I haven't reviewed the PR in detail, but it looks that later, another PR was provided by @Mr-Neutr0n in #3028 that fixes the same issue. As your PR was first, I would treat it preferentially, but I think the solution and the tests in that PR look tighter. Could you please check if you agree with that solution?

One way forward would be to go with the other solution but to treat both of you as co-author. LMK what you think.

[MichalMraz]

Thank you @BenjaminBossan, I checked the solution in PR #3208 and I agree it is tighter and covers more cases. I think treating both of us as co-authors is fair and I'm happy to go with that.

[BenjaminBossan]

Great, thanks @MichalMraz. If @Mr-Neutr0n agrees with this, I would review his PR instead and, when merging, add Michal as co-author.

[Mr-Neutr0n]

I'm more than happy to do so!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[Mr-Neutr0n]

hey @BenjaminBossan please review this PR. thanks!

[BenjaminBossan]

@Mr-Neutr0n I think we agreed to proceed with #3028 instead of this PR. I reviewed that one and am awaiting your updates there.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.