Size-Independent Sample Complexity of Neural Networks

TL;DR

This paper presents new bounds on the sample complexity of neural networks that improve dependence on depth and, under certain conditions, are independent of network size, using novel analytical techniques.

Contribution

It introduces size-independent sample complexity bounds for neural networks based on norm constraints, advancing theoretical understanding of their learning efficiency.

Findings

Complexity bounds improve with depth

Bounds can be independent of network size under certain assumptions

Novel techniques for analyzing Rademacher complexity are developed

Abstract

We study the sample complexity of learning neural networks, by providing new bounds on their Rademacher complexity assuming norm constraints on the parameter matrix of each layer. Compared to previous work, these complexity bounds have improved dependence on the network depth, and under some additional assumptions, are fully independent of the network size (both depth and width). These results are derived using some novel techniques, which may be of independent interest.