Higher Order Generalization Error for First Order Discretization of Langevin Diffusion

TL;DR

This paper analyzes the generalization error of first order discretizations of Langevin diffusion, showing under smoothness assumptions that they can achieve arbitrarily fast convergence rates in terms of iterations.

Contribution

It introduces smoothness conditions under which first order Langevin discretizations attain faster generalization error bounds than previously known.

Findings

First order methods can achieve arbitrarily fast convergence with smoothness assumptions.

The required number of iterations scales as psilon^{-1/N} for any N>0.

Smoothness assumptions enable improved generalization error bounds.

Abstract

We propose a novel approach to analyze generalization error for discretizations of Langevin diffusion, such as the stochastic gradient Langevin dynamics (SGLD). For an $\epsilon$ tolerance of expected generalization error, it is known that a first order discretization can reach this target if we run $\Omega(\epsilon^{-1} \log (\epsilon^{-1}) )$ iterations with $\Omega(\epsilon^{-1})$ samples. In this article, we show that with additional smoothness assumptions, even first order methods can achieve arbitrarily runtime complexity. More precisely, for each $N>0$, we provide a sufficient smoothness condition on the loss function such that a first order discretization can reach $\epsilon$ expected generalization error given $\Omega( \epsilon^{-1/N} \log (\epsilon^{-1}) )$ iterations with $\Omega(\epsilon^{-1})$ samples.