Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations

TL;DR

This paper shows that retraining only the last layer of neural networks can effectively improve robustness to spurious correlations and covariate shifts, offering a simple and computationally efficient alternative to complex methods.

Contribution

It demonstrates that last layer retraining is sufficient for robustness, outperforming or matching state-of-the-art methods with lower complexity.

Findings

Last layer retraining matches or outperforms complex methods on spurious correlation benchmarks.

Retraining on large ImageNet models reduces reliance on background and texture cues.

Significant robustness improvements achieved with only minutes of training on a single GPU.

Abstract

Neural network classifiers can largely rely on simple spurious features, such as backgrounds, to make predictions. However, even in these cases, we show that they still often learn core features associated with the desired attributes of the data, contrary to recent findings. Inspired by this insight, we demonstrate that simple last layer retraining can match or outperform state-of-the-art approaches on spurious correlation benchmarks, but with profoundly lower complexity and computational expenses. Moreover, we show that last layer retraining on large ImageNet-trained models can also significantly reduce reliance on background and texture information, improving robustness to covariate shift, after only minutes of training on a single GPU.