Self-normalized Cram\'er-type Moderate Deviation of Stochastic Gradient Langevin Dynamics

TL;DR

This paper investigates the probabilistic behavior of the empirical measure of stochastic gradient Langevin dynamics (SGLD), deriving a moderate deviation principle and Berry-Esseen bounds using SDE approximation and Stein's method.

Contribution

It introduces a novel analysis of the self-normalized Cramér-type moderate deviation for SGLD and provides Berry-Esseen bounds, advancing theoretical understanding of SGLD's convergence.

Findings

Established Cramér-type moderate deviation results for SGLD

Derived Berry-Esseen bounds for the empirical measure of SGLD

Applied Stein's method to decompose the empirical measure

Abstract

In this paper, we study the self-normalized Cram\'er-type moderate deviation of the empirical measure of the stochastic gradient Langevin dynamics (SGLD). Consequently, we also derive the Berry-Esseen bound for SGLD. Our approach is by constructing a stochastic differential equation (SDE) to approximate the SGLD and then applying Stein's method as developed in [9,19], to decompose the empirical measure into a martingale difference series sum and a negligible remainder term.