Impact of Label Noise on Learning Complex Features

TL;DR

Pre-training neural networks with noisy labels can enhance their ability to learn complex, diverse features and overcome the bias towards simpler decision boundaries, without sacrificing performance.

Contribution

This work demonstrates that noisy label pre-training promotes learning of complex features and diverse representations in neural networks, addressing limitations of traditional regularization methods.

Findings

Pre-training with noisy labels encourages learning complex functions.

Pre-training leads to models capturing broader feature sets.

Performance remains unaffected despite learning more complex features.

Abstract

Neural networks trained with stochastic gradient descent exhibit an inductive bias towards simpler decision boundaries, typically converging to a narrow family of functions, and often fail to capture more complex features. This phenomenon raises concerns about the capacity of deep models to adequately learn and represent real-world datasets. Traditional approaches such as explicit regularization, data augmentation, architectural modifications, etc., have largely proven ineffective in encouraging the models to learn diverse features. In this work, we investigate the impact of pre-training models with noisy labels on the dynamics of SGD across various architectures and datasets. We show that pretraining promotes learning complex functions and diverse features in the presence of noise. Our experiments demonstrate that pre-training with noisy labels encourages gradient descent to find alternate minima that do not solely depend upon simple features, rather learns more complex and broader set of features, without hurting performance.