Some problems about Code

[123456789asdfjkl]

FAN/models/fan.py

Lines 311 to 315 in ee1b7df

	x_res = x
	cls_token = x[:, 0:1]
	cls_token = self.gamma2 * self.mlp(cls_token)
	x = torch.cat([cls_token, x[:, 1:]], dim=1)
	x = x_res + self.drop_path(x)

Hi! Thank you for your great work! According to CaiT, I think the code should be in the following form:
cls_token = x[:, 0:1] + self.drop_path(self.gamma2 * self.mlp(x[:, 0:1] ))
x = torch.cat([cls_token, x[:, 1:], dim=1)

[Youskrpig]

Same question. The forward function in LayerScaleBlockClassAttn realized in timm is:
x = x + self.drop_path(self.gamma1 * self.attn(self.norm1(x)))
x = x + self.drop_path(self.gamma2 * self.mlp(self.norm2(x)))

[zhoudaquan]

FAN/models/fan.py

Lines 311 to 315 in ee1b7df

	x_res = x
	cls_token = x[:, 0:1]
	cls_token = self.gamma2 * self.mlp(cls_token)
	x = torch.cat([cls_token, x[:, 1:]], dim=1)
	x = x_res + self.drop_path(x)

Hi! Thank you for your great work! According to CaiT, I think the code should be in the following form:
cls_token = x[:, 0:1] + self.drop_path(self.gamma2 * self.mlp(x[:, 0:1] ))
x = torch.cat([cls_token, x[:, 1:], dim=1)

Hi, thanks for pointing this out. There are indeed some difference between our implementation and CaiT. But the experiments in the paper are all using the method in the released code…

[zhoudaquan]

Same question. The forward function in LayerScaleBlockClassAttn realized in timm is:
x = x + self.drop_path(self.gamma1 * self.attn(self.norm1(x)))
x = x + self.drop_path(self.gamma2 * self.mlp(self.norm2(x)))

Hi, thanks for pointing this out. There are indeed some difference between our implementation and CaiT. But the experiments in the paper are all using the method in the released code…