A Meta-Analysis of Distributionally-Robust Models

TL;DR

This paper conducts a meta-analysis of recent image classifiers to identify key factors contributing to their robustness against distribution shifts, highlighting the potential of vision-language pre-training.

Contribution

It provides an empirical meta-analysis of recent models, revealing four main commonalities that underpin out-of-distribution robustness, emphasizing the role of vision-language pre-training.

Findings

Identification of four key factors for OOD robustness

Highlighting the importance of vision-language pre-training

Empirical evidence from recent models

Abstract

State-of-the-art image classifiers trained on massive datasets (such as ImageNet) have been shown to be vulnerable to a range of both intentional and incidental distribution shifts. On the other hand, several recent classifiers with favorable out-of-distribution (OOD) robustness properties have emerged, achieving high accuracy on their target tasks while maintaining their in-distribution accuracy on challenging benchmarks. We present a meta-analysis on a wide range of publicly released models, most of which have been published over the last twelve months. Through this meta-analysis, we empirically identify four main commonalities for all the best-performing OOD-robust models, all of which illuminate the considerable promise of vision-language pre-training.