Distributionally Robust Classification on a Data Budget

TL;DR

This paper demonstrates that a standard ResNet-50 trained on a limited dataset can achieve distributional robustness comparable to large-scale models like CLIP trained on hundreds of millions of samples, challenging assumptions about data requirements.

Contribution

Introduces JANuS, a new dataset collection, and shows that robust image classification is possible with significantly less data than previously thought.

Findings

ResNet-50 trained on 2.4 million samples achieves robustness similar to CLIP trained on 400 million samples.

First demonstration of near state-of-the-art robustness with limited data budgets.

Provides datasets and code for reproducibility.

Abstract

Real world uses of deep learning require predictable model behavior under distribution shifts. Models such as CLIP show emergent natural distributional robustness comparable to humans, but may require hundreds of millions of training samples. Can we train robust learners in a domain where data is limited? To rigorously address this question, we introduce JANuS (Joint Annotations and Names Set), a collection of four new training datasets with images, labels, and corresponding captions, and perform a series of carefully controlled investigations of factors contributing to robustness in image classification, then compare those results to findings derived from a large-scale meta-analysis. Using this approach, we show that standard ResNet-50 trained with the cross-entropy loss on 2.4 million image samples can attain comparable robustness to a CLIP ResNet-50 trained on 400 million samples. To our knowledge, this is the first result showing (near) state-of-the-art distributional robustness on limited data budgets. Our dataset is available at \url{https://huggingface.co/datasets/penfever/JANuS_dataset}, and the code used to reproduce our experiments can be found at \url{https://github.com/penfever/vlhub/}.