Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision

TL;DR

This paper investigates how the choice of random seed affects the accuracy of deep learning models in computer vision, revealing significant variability and the potential for outliers that outperform or underperform average results.

Contribution

It provides a large-scale analysis of seed variability on CIFAR-10 and ImageNet, highlighting the impact of seed selection on model performance in vision tasks.

Findings

Seed choice can cause significant accuracy variation.

Outliers with much better or worse performance are common.

Large-scale seed scans reveal the importance of seed selection.

Abstract

In this paper I investigate the effect of random seed selection on the accuracy when using popular deep learning architectures for computer vision. I scan a large amount of seeds (up to $10^4$) on CIFAR 10 and I also scan fewer seeds on Imagenet using pre-trained models to investigate large scale datasets. The conclusions are that even if the variance is not very large, it is surprisingly easy to find an outlier that performs much better or much worse than the average.