Models Out of Line: A Fourier Lens on Distribution Shift Robustness

TL;DR

This paper investigates the spectral properties of models and data that influence out-of-distribution robustness, using a Fourier perspective, and introduces RobustNets, a set of pretrained models with varying OOD robustness levels.

Contribution

It provides a comprehensive empirical study of OOD robustness, analyzes spectral properties through a Fourier lens, and introduces RobustNets with different robustness levels.

Findings

Spectral properties influence OOD robustness.

CLIP models show notable spectral robustness.

No single metric fully explains OOD robustness.

Abstract

Improving the accuracy of deep neural networks (DNNs) on out-of-distribution (OOD) data is critical to an acceptance of deep learning (DL) in real world applications. It has been observed that accuracies on in-distribution (ID) versus OOD data follow a linear trend and models that outperform this baseline are exceptionally rare (and referred to as "effectively robust"). Recently, some promising approaches have been developed to improve OOD robustness: model pruning, data augmentation, and ensembling or zero-shot evaluating large pretrained models. However, there still is no clear understanding of the conditions on OOD data and model properties that are required to observe effective robustness. We approach this issue by conducting a comprehensive empirical study of diverse approaches that are known to impact OOD robustness on a broad range of natural and synthetic distribution shifts of CIFAR-10 and ImageNet. In particular, we view the "effective robustness puzzle" through a Fourier lens and ask how spectral properties of both models and OOD data influence the corresponding effective robustness. We find this Fourier lens offers some insight into why certain robust models, particularly those from the CLIP family, achieve OOD robustness. However, our analysis also makes clear that no known metric is consistently the best explanation (or even a strong explanation) of OOD robustness. Thus, to aid future research into the OOD puzzle, we address the gap in publicly-available models with effective robustness by introducing a set of pretrained models--RobustNets--with varying levels of OOD robustness.