Control+Shift: Generating Controllable Distribution Shifts

TL;DR

This paper introduces a method to generate datasets with controllable distribution shifts to analyze how different models degrade in performance, revealing that larger datasets and stronger biases can improve robustness.

Contribution

The paper presents a novel approach to systematically generate and analyze distribution shifts using generative models, enabling detailed robustness studies.

Findings

Model performance declines with increasing shift intensity.

Data augmentation does not prevent performance degradation.

Larger datasets beyond a point do not improve robustness.

Abstract

We propose a new method for generating realistic datasets with distribution shifts using any decoder-based generative model. Our approach systematically creates datasets with varying intensities of distribution shifts, facilitating a comprehensive analysis of model performance degradation. We then use these generated datasets to evaluate the performance of various commonly used networks and observe a consistent decline in performance with increasing shift intensity, even when the effect is almost perceptually unnoticeable to the human eye. We see this degradation even when using data augmentations. We also find that enlarging the training dataset beyond a certain point has no effect on the robustness and that stronger inductive biases increase robustness.