On Random Subset Generalization Error Bounds and the Stochastic Gradient Langevin Dynamics Algorithm

TL;DR

This paper unifies and extends generalization error bounds for stochastic gradient Langevin dynamics, revealing limitations and providing refined bounds for complex loss functions, advancing theoretical understanding in this area.

Contribution

It introduces new bounds for stochastic gradient Langevin dynamics and refines existing ones for large gradient norms, unifying various prior bounds within a single framework.

Findings

Unified several expected generalization error bounds using a common framework.

Extended bounds to stochastic gradient Langevin dynamics and refined them for large gradients.

Identified limitations in existing bounds and provided more comprehensive theoretical guarantees.

Abstract

In this work, we unify several expected generalization error bounds based on random subsets using the framework developed by Hellstr\"om and Durisi [1]. First, we recover the bounds based on the individual sample mutual information from Bu et al. [2] and on a random subset of the dataset from Negrea et al. [3]. Then, we introduce their new, analogous bounds in the randomized subsample setting from Steinke and Zakynthinou [4], and we identify some limitations of the framework. Finally, we extend the bounds from Haghifam et al. [5] for Langevin dynamics to stochastic gradient Langevin dynamics and we refine them for loss functions with potentially large gradient norms.