DeepSpeed compatibility (?)

[NZ42]

I'm interested in experimenting with modula for h-param transfer, and I was wondering whether I should worry about incompatibility issues with ZeRO (particularly, stage 2)? Thanks!

[jxbz]

Hi @NZ42! We haven't implemented things like model sharding yet. I would imagine you may run into compatibility issues with DeepSpeed, and at the moment it would require a little work to combine DeepSpeed and modula. This is because we wrote our own class to store model weights: https://github.com/jxbz/modula/blob/main/modula/vector.py and so this may need to be modified.

If you do try I'd be happy to try to help / answer questions. I'll be much freer after this coming weekend.