Robust Learning with Progressive Data Expansion Against Spurious Correlation

TL;DR

This paper introduces PDE, a training algorithm that progressively expands data groups to improve model robustness against spurious correlations, achieving better worst-group accuracy and faster training on various datasets.

Contribution

The paper provides a theoretical analysis of nonlinear CNNs learning spurious features and proposes PDE, a novel data expansion method that enhances robustness and efficiency.

Findings

PDE improves worst-group accuracy by 2.8% over state-of-the-art.

PDE achieves up to 10x faster training efficiency.

Experimental results on synthetic and real datasets validate PDE's effectiveness.

Abstract

While deep learning models have shown remarkable performance in various tasks, they are susceptible to learning non-generalizable spurious features rather than the core features that are genuinely correlated to the true label. In this paper, beyond existing analyses of linear models, we theoretically examine the learning process of a two-layer nonlinear convolutional neural network in the presence of spurious features. Our analysis suggests that imbalanced data groups and easily learnable spurious features can lead to the dominance of spurious features during the learning process. In light of this, we propose a new training algorithm called PDE that efficiently enhances the model's robustness for a better worst-group performance. PDE begins with a group-balanced subset of training data and progressively expands it to facilitate the learning of the core features. Experiments on synthetic and real-world benchmark datasets confirm the superior performance of our method on models such as ResNets and Transformers. On average, our method achieves a 2.8% improvement in worst-group accuracy compared with the state-of-the-art method, while enjoying up to 10x faster training efficiency. Codes are available at https://github.com/uclaml/PDE.