Error Bounds of Supervised Classification from Information-Theoretic Perspective

TL;DR

This paper derives information-theoretic bounds on the expected risk of deep neural network classifiers, linking model complexity, distribution smoothness, and sample size to generalization, and validates these bounds empirically.

Contribution

It introduces new bounds on fitting error and generalization error, providing theoretical insights into overparameterization and flat minima in deep learning.

Findings

Theoretical bounds correlate with actual expected risk.

Fitting error bound relates gradient and parameter count.

Generalization error is bounded by distribution smoothness and sample size.

Abstract

In this paper, we explore bounds on the expected risk when using deep neural networks for supervised classification from an information theoretic perspective. Firstly, we introduce model risk and fitting error, which are derived from further decomposing the empirical risk. Model risk represents the expected value of the loss under the model's predicted probabilities and is exclusively dependent on the model. Fitting error measures the disparity between the empirical risk and model risk. Then, we derive the upper bound on fitting error, which links the back-propagated gradient and the model's parameter count with the fitting error. Furthermore, we demonstrate that the generalization errors are bounded by the classification uncertainty, which is characterized by both the smoothness of the distribution and the sample size. Based on the bounds on fitting error and generalization, by utilizing the triangle inequality, we establish an upper bound on the expected risk. This bound is applied to provide theoretical explanations for overparameterization, non-convex optimization and flat minima in deep learning. Finally, empirical verification confirms a significant positive correlation between the derived theoretical bounds and the practical expected risk, thereby affirming the practical relevance of the theoretical findings.