Information-Theoretic Generalization Bounds for Deep Neural Networks

TL;DR

This paper develops information-theoretic bounds on the generalization error of deep neural networks, revealing how depth and layer-wise information contraction influence generalization performance.

Contribution

It introduces hierarchical bounds based on KL divergence and Wasserstein distance, and analyzes information contraction across layers for regularized DNNs, providing new insights into depth's role.

Findings

Deeper layers can serve as a generalization funnel with minimal Wasserstein distance.

Generalization bounds tighten as the network depth increases under certain conditions.

Deeper and narrower networks may generalize better in specific settings.

Abstract

Deep neural networks (DNNs) exhibit an exceptional capacity for generalization in practical applications. This work aims to capture the effect and benefits of depth for supervised learning via information-theoretic generalization bounds. We first derive two hierarchical bounds on the generalization error in terms of the Kullback-Leibler (KL) divergence or the 1-Wasserstein distance between the train and test distributions of the network internal representations. The KL divergence bound shrinks as the layer index increases, while the Wasserstein bound implies the existence of a layer that serves as a generalization funnel, which attains a minimal 1-Wasserstein distance. Analytic expressions for both bounds are derived under the setting of binary Gaussian classification with linear DNNs. To quantify the contraction of the relevant information measures when moving deeper into the network, we analyze the strong data processing inequality (SDPI) coefficient between consecutive layers of three regularized DNN models: $\mathsf{Dropout}$, $\mathsf{DropConnect}$, and Gaussian noise injection. This enables refining our generalization bounds to capture the contraction as a function of the network architecture parameters. Specializing our results to DNNs with a finite parameter space and the Gibbs algorithm reveals that deeper yet narrower network architectures generalize better in those examples, although how broadly this statement applies remains a question.