Could calflops can be used to measure Mamba?

[RICKand-MORTY]

Hi! I want to measure the FLOPs and MACs of the Mamba model, but I found that some packages, such as thop and ptflops, cannot measure the Mamba model. Could calflops measure the FLOPs and MACs of the Mamba model?

I have tried using calflops to measure the Mamba model, but the results I obtained are as follows:

(0): Mamba( 552 = 0.0928% Params, 51.22 MMACs = 1.1699% MACs, 102.45 MFLOPS = 1.1471% FLOPs (in_proj): Linear(64 = 0.0108% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=4, out_features=16, bias=False) (conv1d): Conv1d(40 = 0.0067% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, 8, 8, kernel_size=(4,), stride=(1,), padding=(3,), groups=8) (act): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs) (x_proj): Linear(264 = 0.0444% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=8, out_features=33, bias=False) (dt_proj): Linear(16 = 0.0027% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=1, out_features=8, bias=True) (out_proj): Linear(32 = 0.0054% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=8, out_features=4, bias=False) )

I found that the MACs and FLOPs of in_proj, conv1d, x_proj, dt_proj, and out_proj inside Mamba are all zero. Is this normal, or is it because calflops cannot compute the MACs and FLOPs of these layers? I am looking forward to your response!

[Devin-Pi]

Hi! I want to measure the FLOPs and MACs of the Mamba model, but I found that some packages, such as thop and ptflops, cannot measure the Mamba model. Could calflops measure the FLOPs and MACs of the Mamba model?

I have tried using calflops to measure the Mamba model, but the results I obtained are as follows:

(0): Mamba( 552 = 0.0928% Params, 51.22 MMACs = 1.1699% MACs, 102.45 MFLOPS = 1.1471% FLOPs (in_proj): Linear(64 = 0.0108% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=4, out_features=16, bias=False) (conv1d): Conv1d(40 = 0.0067% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, 8, 8, kernel_size=(4,), stride=(1,), padding=(3,), groups=8) (act): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs) (x_proj): Linear(264 = 0.0444% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=8, out_features=33, bias=False) (dt_proj): Linear(16 = 0.0027% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=1, out_features=8, bias=True) (out_proj): Linear(32 = 0.0054% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=8, out_features=4, bias=False) )

I found that the MACs and FLOPs of in_proj, conv1d, x_proj, dt_proj, and out_proj inside Mamba are all zero. Is this normal, or is it because calflops cannot compute the MACs and FLOPs of these layers? I am looking forward to your response!

Hi, Morty! Have you found any ways to measure the FLOPs of mamba?

[RICKand-MORTY]

Hi! I want to measure the FLOPs and MACs of the Mamba model, but I found that some packages, such as thop and ptflops, cannot measure the Mamba model. Could calflops measure the FLOPs and MACs of the Mamba model?
I have tried using calflops to measure the Mamba model, but the results I obtained are as follows:
(0): Mamba( 552 = 0.0928% Params, 51.22 MMACs = 1.1699% MACs, 102.45 MFLOPS = 1.1471% FLOPs (in_proj): Linear(64 = 0.0108% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=4, out_features=16, bias=False) (conv1d): Conv1d(40 = 0.0067% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, 8, 8, kernel_size=(4,), stride=(1,), padding=(3,), groups=8) (act): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs) (x_proj): Linear(264 = 0.0444% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=8, out_features=33, bias=False) (dt_proj): Linear(16 = 0.0027% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=1, out_features=8, bias=True) (out_proj): Linear(32 = 0.0054% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=8, out_features=4, bias=False) )
I found that the MACs and FLOPs of in_proj, conv1d, x_proj, dt_proj, and out_proj inside Mamba are all zero. Is this normal, or is it because calflops cannot compute the MACs and FLOPs of these layers? I am looking forward to your response!

Hi, Morty! Have you found any ways to measure the FLOPs of mamba?

Sorry, so far, I haven’t found a suitable tool that can fully measure the computational burden of Mamba-based models. However, I’ve reported this issue to the author of TorchMeter (TorchMeter/torchmeter#27), who responded that atomic-level computational behavior awareness will be adopted in the next iteration.

[Devin-Pi]

Hi! I want to measure the FLOPs and MACs of the Mamba model, but I found that some packages, such as thop and ptflops, cannot measure the Mamba model. Could calflops measure the FLOPs and MACs of the Mamba model?
I have tried using calflops to measure the Mamba model, but the results I obtained are as follows:
(0): Mamba( 552 = 0.0928% Params, 51.22 MMACs = 1.1699% MACs, 102.45 MFLOPS = 1.1471% FLOPs (in_proj): Linear(64 = 0.0108% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=4, out_features=16, bias=False) (conv1d): Conv1d(40 = 0.0067% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, 8, 8, kernel_size=(4,), stride=(1,), padding=(3,), groups=8) (act): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs) (x_proj): Linear(264 = 0.0444% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=8, out_features=33, bias=False) (dt_proj): Linear(16 = 0.0027% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=1, out_features=8, bias=True) (out_proj): Linear(32 = 0.0054% Params, 0 MACs = 0% MACs, 0 FLOPS = 0% FLOPs, in_features=8, out_features=4, bias=False) )
I found that the MACs and FLOPs of in_proj, conv1d, x_proj, dt_proj, and out_proj inside Mamba are all zero. Is this normal, or is it because calflops cannot compute the MACs and FLOPs of these layers? I am looking forward to your response!

Hi, Morty! Have you found any ways to measure the FLOPs of mamba?

Sorry, so far, I haven’t found a suitable tool that can fully measure the computational burden of Mamba-based models. However, I’ve reported this issue to the author of TorchMeter (TorchMeter/torchmeter#27), who responded that atomic-level computational behavior awareness will be adopted in the next iteration.

Thank you!