Feature contamination: Neural networks learn uncorrelated features and fail to generalize

TL;DR

This paper investigates why neural networks fail to generalize under distribution shifts, revealing a phenomenon called feature contamination where uncorrelated features are learned alongside predictive ones, impairing robustness.

Contribution

It provides the first empirical and theoretical analysis of feature contamination, showing its role in out-of-distribution generalization failure in neural networks.

Findings

Neural networks can learn uncorrelated features with predictive ones, leading to failure under distribution shifts.

Explicitly fitting a teacher network's representations does not guarantee out-of-distribution generalization.

Feature contamination differs from spurious correlation explanations in the literature.

Abstract

Learning representations that generalize under distribution shifts is critical for building robust machine learning models. However, despite significant efforts in recent years, algorithmic advances in this direction have been limited. In this work, we seek to understand the fundamental difficulty of out-of-distribution generalization with deep neural networks. We first empirically show that perhaps surprisingly, even allowing a neural network to explicitly fit the representations obtained from a teacher network that can generalize out-of-distribution is insufficient for the generalization of the student network. Then, by a theoretical study of two-layer ReLU networks optimized by stochastic gradient descent (SGD) under a structured feature model, we identify a fundamental yet unexplored feature learning proclivity of neural networks, feature contamination: neural networks can learn uncorrelated features together with predictive features, resulting in generalization failure under distribution shifts. Notably, this mechanism essentially differs from the prevailing narrative in the literature that attributes the generalization failure to spurious correlations. Overall, our results offer new insights into the non-linear feature learning dynamics of neural networks and highlight the necessity of considering inductive biases in out-of-distribution generalization.