reproducibility

[mehrnia]

Hi,
I just want to bring an attention that. when trained the resnet18 wit same parameters that was mentioned in the paper, I get different ACC and AUC.
Can anyone give me an insight into what approach I should follow?
I have attached the results of 10 independent runs as well.

code:
data_flag = 'breastmnist'
download = True
NUM_EPOCHS = 100
BATCH_SIZE = 128
lr = 0.001
im_size=224
from torchvision.models import resnet18
import timm
device='cuda'
model = resnet18(num_classes=n_classes).cuda()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=lr)
scheduler = optim.lr_scheduler.MultiStepLR(optimizer, milestones=[50, 75], gamma=0.1)

The result for im_size=224 is also different.

[duducheng]

Hi @mehrnia ,

Thanks for sharing the results of your repeated runs!

Indeed, BreastMNIST is one of the smallest datasets in the MedMNIST collection, and as such, it's particularly prone to performance variability due to limited sample size. This means that even with the same model architecture and training configuration, metrics like ACC and AUC can fluctuate more significantly compared to larger datasets.

In fact, the numbers you've reported already seem quite close to those from our original benchmark (which were based on averaging over 3 runs). Given the nature of the dataset, we actually encourage users to repeat experiments as needed for their own research purposes, especially when targeting statistical robustness.

This kind of variability itself might even pose interesting scientific questions, such as exploring model stability under data scarcity or evaluating different techniques for variance reduction.

Hope this helps!

JC