Towards A Unified PAC-Bayesian Framework for Norm-based Generalization Bounds

TL;DR

This paper introduces a unified PAC-Bayesian framework for deriving norm-based generalization bounds in deep learning, addressing previous limitations by incorporating structured weight perturbations and network sensitivities.

Contribution

It reformulates generalization bounds as a stochastic optimization over anisotropic Gaussian posteriors, enabling architecture-aware and tighter bounds.

Findings

Recover several existing PAC-Bayesian bounds as special cases

Derive bounds comparable to or tighter than state-of-the-art

Incorporate heterogeneous parameter sensitivities and architecture structures

Abstract

Understanding the generalization behavior of deep neural networks remains a fundamental challenge in modern statistical learning theory. Among existing approaches, PAC-Bayesian norm-based bounds have demonstrated particular promise due to their data-dependent nature and their ability to capture algorithmic and geometric properties of learned models. However, most existing results rely on isotropic Gaussian posteriors, heavy use of spectral-norm concentration for weight perturbations, and largely architecture-agnostic analyses, which together limit both the tightness and practical relevance of the resulting bounds. To address these limitations, in this work, we propose a unified framework for PAC-Bayesian norm-based generalization by reformulating the derivation of generalization bounds as a stochastic optimization problem over anisotropic Gaussian posteriors. The key to our approach is a sensitivity matrix that quantifies the network outputs with respect to structured weight perturbations, enabling the explicit incorporation of heterogeneous parameter sensitivities and architectural structures. By imposing different structural assumptions on this sensitivity matrix, we derive a family of generalization bounds that recover several existing PAC-Bayesian results as special cases, while yielding bounds that are comparable to or tighter than state-of-the-art approaches. Such a unified framework provides a principled and flexible way for geometry-/structure-aware and interpretable generalization analysis in deep learning.