A Generalization Bound for Nearly-Linear Networks

TL;DR

This paper introduces a priori generalization bounds for nearly-linear neural networks, which are non-vacuous and do not require training data, advancing theoretical understanding of neural network generalization.

Contribution

It presents the first non-vacuous a-priori generalization bounds for neural networks close to linear, based on their perturbation from linearity.

Findings

Bounds are non-vacuous for nearly-linear networks

Bounds do not require actual training data for evaluation

First such bounds for this class of neural networks

Abstract

We consider nonlinear networks as perturbations of linear ones. Based on this approach, we present novel generalization bounds that become non-vacuous for networks that are close to being linear. The main advantage over the previous works which propose non-vacuous generalization bounds is that our bounds are a-priori: performing the actual training is not required for evaluating the bounds. To the best of our knowledge, they are the first non-vacuous generalization bounds for neural nets possessing this property.