LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models

TL;DR

This paper introduces LAION-C, a new OOD benchmark with six novel distortions designed for web-scale datasets, revealing current models' limitations and a shift in OOD robustness from humans to models.

Contribution

The paper presents LAION-C, a novel OOD benchmark with distortions tailored for web-scale datasets, and provides comprehensive evaluations showing models' challenges and a paradigm shift in robustness.

Findings

LAION-C significantly challenges state-of-the-art models.

Models now match or outperform humans in OOD robustness.

LAION-C reveals limitations of existing benchmarks for web-scale datasets.

Abstract

Out-of-distribution (OOD) robustness is a desired property of computer vision models. Improving model robustness requires high-quality signals from robustness benchmarks to quantify progress. While various benchmark datasets such as ImageNet-C were proposed in the ImageNet era, most ImageNet-C corruption types are no longer OOD relative to today's large, web-scraped datasets, which already contain common corruptions such as blur or JPEG compression artifacts. Consequently, these benchmarks are no longer well-suited for evaluating OOD robustness in the era of web-scale datasets. Indeed, recent models show saturating scores on ImageNet-era OOD benchmarks, indicating that it is unclear whether models trained on web-scale datasets truly become better at OOD generalization or whether they have simply been exposed to the test distortions during training. To address this, we introduce LAION-C as a benchmark alternative for ImageNet-C. LAION-C consists of six novel distortion types specifically designed to be OOD, even for web-scale datasets such as LAION. In a comprehensive evaluation of state-of-the-art models, we find that the LAION-C dataset poses significant challenges to contemporary models, including MLLMs such as Gemini and GPT-4o. We additionally conducted a psychophysical experiment to evaluate the difficulty of our corruptions for human observers, enabling a comparison of models to lab-quality human robustness data. We observe a paradigm shift in OOD generalization: from humans outperforming models, to the best models now matching or outperforming the best human observers.