Benign Overfitting in Two-Layer ReLU Convolutional Neural Networks for XOR Data

TL;DR

This paper demonstrates that over-parameterized two-layer ReLU CNNs can effectively learn XOR classification tasks with label noise, achieving near-optimal accuracy under certain conditions, and provides theoretical bounds on their performance.

Contribution

It offers the first theoretical analysis of CNNs learning XOR-type problems with noise, establishing conditions for near-optimal learning and bounds when these conditions are not met.

Findings

CNNs can learn XOR problems with high accuracy under specific conditions.

A lower bound shows CNNs' performance is limited when conditions are not satisfied.

The study extends understanding of benign overfitting in nonlinear models.

Abstract

Modern deep learning models are usually highly over-parameterized so that they can overfit the training data. Surprisingly, such overfitting neural networks can usually still achieve high prediction accuracy. To study this "benign overfitting" phenomenon, a line of recent works has theoretically studied the learning of linear models and two-layer neural networks. However, most of these analyses are still limited to the very simple learning problems where the Bayes-optimal classifier is linear. In this work, we investigate a class of XOR-type classification tasks with label-flipping noises. We show that, under a certain condition on the sample complexity and signal-to-noise ratio, an over-parameterized ReLU CNN trained by gradient descent can achieve near Bayes-optimal accuracy. Moreover, we also establish a matching lower bound result showing that when the previous condition is not satisfied, the prediction accuracy of the obtained CNN is an absolute constant away from the Bayes-optimal rate. Our result demonstrates that CNNs have a remarkable capacity to efficiently learn XOR problems, even in the presence of highly correlated features.