Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience

TL;DR

This paper introduces a novel PAC-Bayesian framework that provides deterministic generalization bounds for deep networks by leveraging noise-resilience and flat minima properties, without relying on stochastic or compressed parameters.

Contribution

It develops a new PAC-Bayesian approach that generalizes flat minima conditions from training to test data for deterministic deep networks, improving existing generalization guarantees.

Findings

Provides a bound that does not scale with spectral norms of weights.

Shows training interactions imply test interactions under certain conditions.

Guarantees apply to original deterministic networks, not just stochastic or compressed ones.

Abstract

The ability of overparameterized deep networks to generalize well has been linked to the fact that stochastic gradient descent (SGD) finds solutions that lie in flat, wide minima in the training loss -- minima where the output of the network is resilient to small random noise added to its parameters. So far this observation has been used to provide generalization guarantees only for neural networks whose parameters are either \textit{stochastic} or \textit{compressed}. In this work, we present a general PAC-Bayesian framework that leverages this observation to provide a bound on the original network learned -- a network that is deterministic and uncompressed. What enables us to do this is a key novelty in our approach: our framework allows us to show that if on training data, the interactions between the weight matrices satisfy certain conditions that imply a wide training loss minimum, these conditions themselves {\em generalize} to the interactions between the matrices on test data, thereby implying a wide test loss minimum. We then apply our general framework in a setup where we assume that the pre-activation values of the network are not too small (although we assume this only on the training data). In this setup, we provide a generalization guarantee for the original (deterministic, uncompressed) network, that does not scale with product of the spectral norms of the weight matrices -- a guarantee that would not have been possible with prior approaches.