Benign Overfitting in Two-layer Convolutional Neural Networks

TL;DR

This paper investigates the conditions under which benign overfitting occurs in two-layer CNNs, revealing a phase transition driven by the signal-to-noise ratio and providing the first precise characterization of this phenomenon in CNNs.

Contribution

It provides the first theoretical analysis of benign overfitting in two-layer CNNs, identifying a phase transition based on the signal-to-noise ratio.

Findings

When the signal-to-noise ratio is high, CNNs trained by gradient descent achieve near-zero training and test loss.

A sharp phase transition exists between benign and harmful overfitting, depending on the signal-to-noise ratio.

The study offers the first precise conditions for benign overfitting in convolutional neural networks.

Abstract

Modern neural networks often have great expressive power and can be trained to overfit the training data, while still achieving a good test performance. This phenomenon is referred to as "benign overfitting". Recently, there emerges a line of works studying "benign overfitting" from the theoretical perspective. However, they are limited to linear models or kernel/random feature models, and there is still a lack of theoretical understanding about when and how benign overfitting occurs in neural networks. In this paper, we study the benign overfitting phenomenon in training a two-layer convolutional neural network (CNN). We show that when the signal-to-noise ratio satisfies a certain condition, a two-layer CNN trained by gradient descent can achieve arbitrarily small training and test loss. On the other hand, when this condition does not hold, overfitting becomes harmful and the obtained CNN can only achieve a constant level test loss. These together demonstrate a sharp phase transition between benign overfitting and harmful overfitting, driven by the signal-to-noise ratio. To the best of our knowledge, this is the first work that precisely characterizes the conditions under which benign overfitting can occur in training convolutional neural networks.