On Data Scaling in Masked Image Modeling

TL;DR

This paper demonstrates that masked image modeling benefits from larger datasets and longer training, challenging the belief that it cannot scale with more data, and introduces validation loss as a predictor of downstream performance.

Contribution

The study provides extensive empirical evidence that masked image modeling scales effectively with data size and training duration, and proposes validation loss as a pre-training performance indicator.

Findings

Large models overfit small datasets

Longer training improves performance with more data

Validation loss predicts downstream task success

Abstract

An important goal of self-supervised learning is to enable model pre-training to benefit from almost unlimited data. However, one method that has recently become popular, namely masked image modeling (MIM), is suspected to be unable to benefit from larger data. In this work, we break this misconception through extensive experiments, with data scales ranging from 10\% of ImageNet-1K to full ImageNet-22K, model sizes ranging from 49 million to 1 billion, and training lengths ranging from 125K iterations to 500K iterations. Our study reveals that: (i) Masked image modeling is also demanding on larger data. We observed that very large models got over-fitted with relatively small data; (ii) The length of training matters. Large models trained with masked image modeling can benefit from more data with longer training; (iii) The validation loss in pre-training is a good indicator to measure how well the model performs for fine-tuning on multiple tasks. This observation allows us to pre-evaluate pre-trained models in advance without having to make costly trial-and-error assessments of downstream tasks. We hope that our findings will advance the understanding of masked image modeling in terms of scaling ability.