Investigating the Impact of Model Width and Density on Generalization in Presence of Label Noise

TL;DR

This paper explores how label noise affects the generalization of overparameterized neural networks, revealing a final ascent in test loss and showing that reducing model density can improve robustness under noisy labels.

Contribution

It uncovers the final ascent phenomenon caused by label noise, provides a theoretical explanation, and demonstrates that reducing model density enhances generalization in noisy settings.

Findings

Label noise causes a final ascent in the double descent test loss curve.

Reducing model density improves generalization under label noise.

Larger regularization and robust methods can worsen the final ascent.

Abstract

Increasing the size of overparameterized neural networks has been a key in achieving state-of-the-art performance. This is captured by the double descent phenomenon, where the test loss follows a decreasing-increasing-decreasing pattern (or sometimes monotonically decreasing) as model width increases. However, the effect of label noise on the test loss curve has not been fully explored. In this work, we uncover an intriguing phenomenon where label noise leads to a \textit{final ascent} in the originally observed double descent curve. Specifically, under a sufficiently large noise-to-sample-size ratio, optimal generalization is achieved at intermediate widths. Through theoretical analysis, we attribute this phenomenon to the shape transition of test loss variance induced by label noise. Furthermore, we extend the final ascent phenomenon to model density and provide the first theoretical characterization showing that reducing density by randomly dropping trainable parameters improves generalization under label noise. We also thoroughly examine the roles of regularization and sample size. Surprisingly, we find that larger $\ell_2$ regularization and robust learning methods against label noise exacerbate the final ascent. We confirm the validity of our findings through extensive experiments on ReLu networks trained on MNIST, ResNets/ViTs trained on CIFAR-10/100, and InceptionResNet-v2 trained on Stanford Cars with real-world noisy labels.