Regularization by Misclassification in ReLU Neural Networks

TL;DR

This paper investigates how label noise during training causes ReLU neural networks to develop sparse activations and potentially improve test performance, supported by both experiments and theoretical analysis.

Contribution

It introduces a novel perspective on implicit regularization via label noise and provides a theoretical understanding of sparsification mechanisms in ReLU networks.

Findings

Label noise promotes sparsity in network activations.

Moderate label noise can reduce test error.

Extreme noise leads to network withering, depending on training conditions.

Abstract

We study the implicit bias of ReLU neural networks trained by a variant of SGD where at each step, the label is changed with probability $p$ to a random label (label smoothing being a close variant of this procedure). Our experiments demonstrate that label noise propels the network to a sparse solution in the following sense: for a typical input, a small fraction of neurons are active, and the firing pattern of the hidden layers is sparser. In fact, for some instances, an appropriate amount of label noise does not only sparsify the network but further reduces the test error. We then turn to the theoretical analysis of such sparsification mechanisms, focusing on the extremal case of $p=1$. We show that in this case, the network withers as anticipated from experiments, but surprisingly, in different ways that depend on the learning rate and the presence of bias, with either weights vanishing or neurons ceasing to fire.