Compression Implies Generalization

TL;DR

This paper introduces a new compression-based framework that extends generalization bounds from compressed neural networks to the original models, also applying to SVMs and Boosting, advancing theoretical understanding of model generalization.

Contribution

The authors develop a simple, powerful compression framework that extends existing bounds to uncompressed networks and applies to other models like SVMs and Boosting.

Findings

Framework extends generalization bounds to original neural networks.

Framework provides simple proofs for bounds on SVMs and Boosting.

Supports the hypothesis that compression relates to generalization performance.

Abstract

Explaining the surprising generalization performance of deep neural networks is an active and important line of research in theoretical machine learning. Influential work by Arora et al. (ICML'18) showed that, noise stability properties of deep nets occurring in practice can be used to provably compress model representations. They then argued that the small representations of compressed networks imply good generalization performance albeit only of the compressed nets. Extending their compression framework to yield generalization bounds for the original uncompressed networks remains elusive. Our main contribution is the establishment of a compression-based framework for proving generalization bounds. The framework is simple and powerful enough to extend the generalization bounds by Arora et al. to also hold for the original network. To demonstrate the flexibility of the framework, we also show that it allows us to give simple proofs of the strongest known generalization bounds for other popular machine learning models, namely Support Vector Machines and Boosting.