Leveraging Flatness to Improve Information-Theoretic Generalization Bounds for SGD

TL;DR

This paper introduces a new information-theoretic generalization bound that leverages flatness in SGD, resulting in tighter bounds that better reflect the model's generalization performance, especially in deep neural networks.

Contribution

It derives a flatness-aware IT bound for SGD, improving the understanding of how flatness influences generalization and providing tighter, more accurate bounds.

Findings

Bound reflects improved flatness correlates with better generalization.

Experiments show the bound is numerically tighter and aligns with empirical results.

Application to convex problems improves convergence rates from Ω(1) to O(1/√n).

Abstract

Information-theoretic (IT) generalization bounds have been used to study the generalization of learning algorithms. These bounds are intrinsically data- and algorithm-dependent so that one can exploit the properties of data and algorithm to derive tighter bounds. However, we observe that although the flatness bias is crucial for SGD's generalization, these bounds fail to capture the improved generalization under better flatness and are also numerically loose. This is caused by the inadequate leverage of SGD's flatness bias in existing IT bounds. This paper derives a more flatness-leveraging IT bound for the flatness-favoring SGD. The bound indicates the learned models generalize better if the large-variance directions of the final weight covariance have small local curvatures in the loss landscape. Experiments on deep neural networks show our bound not only correctly reflects the better generalization when flatness is improved, but is also numerically much tighter. This is achieved by a flexible technique called "omniscient trajectory". When applied to Gradient Descent's minimax excess risk on convex-Lipschitz-Bounded problems, it improves representative IT bounds' $\Omega(1)$ rates to $O(1/\sqrt{n})$. It also implies a by-pass of memorization-generalization trade-offs.