Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks

TL;DR

This paper introduces a new complexity measure based on unit-wise capacities to better understand why over-parametrized neural networks tend to generalize well, providing tighter bounds and empirical correlation with test error.

Contribution

It proposes a novel capacity measure for two-layer ReLU networks and offers a matching lower bound for Rademacher complexity, advancing theoretical understanding of over-parametrization effects.

Findings

Capacity measure correlates with test error behavior

Provides tighter generalization bounds for neural networks

Establishes a lower bound for Rademacher complexity

Abstract

Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization. In this work we suggest a novel complexity measure based on unit-wise capacities resulting in a tighter generalization bound for two layer ReLU networks. Our capacity bound correlates with the behavior of test error with increasing network sizes, and could potentially explain the improvement in generalization with over-parametrization. We further present a matching lower bound for the Rademacher complexity that improves over previous capacity lower bounds for neural networks.