Generalization bounds for regression and classification on adaptive covering input domains

TL;DR

This paper derives generalization bounds for regression and classification tasks on adaptive input domains, highlighting the impact of network parameters and architecture on learning efficiency and overfitting.

Contribution

It provides novel bounds that relate the generalization error to network parameters and architecture, emphasizing the benefits of over-parameterization.

Findings

Generalization bounds are inversely proportional to a polynomial of network parameters.

Regression and classification bounds differ in sample complexity requirements.

Over-parameterized networks can achieve benign overfitting under certain conditions.

Abstract

Our main focus is on the generalization bound, which serves as an upper limit for the generalization error. Our analysis delves into regression and classification tasks separately to ensure a thorough examination. We assume the target function is real-valued and Lipschitz continuous for regression tasks. We use the 2-norm and a root-mean-square-error (RMSE) variant to measure the disparities between predictions and actual values. In the case of classification tasks, we treat the target function as a one-hot classifier, representing a piece-wise constant function, and employ 0/1 loss for error measurement. Our analysis underscores the differing sample complexity required to achieve a concentration inequality of generalization bounds, highlighting the variation in learning efficiency for regression and classification tasks. Furthermore, we demonstrate that the generalization bounds for regression and classification functions are inversely proportional to a polynomial of the number of parameters in a network, with the degree depending on the hypothesis class and the network architecture. These findings emphasize the advantages of over-parameterized networks and elucidate the conditions for benign overfitting in such systems.