Generalization Bounds for Stochastic Gradient Langevin Dynamics: A Unified View via Information Leakage Analysis

TL;DR

This paper offers a unified theoretical and empirical analysis of the generalization bounds of SGLD, using information leakage as a key perspective, and explains prior findings on membership privacy.

Contribution

It introduces a unified framework based on privacy leakage analysis to derive and explain generalization bounds of SGLD, connecting multiple theoretical approaches.

Findings

Unified theoretical framework for SGLD generalization bounds

Empirical evidence of information leakage in SGLD

Explanation of membership privacy in prior SGLD studies

Abstract

Recently, generalization bounds of the non-convex empirical risk minimization paradigm using Stochastic Gradient Langevin Dynamics (SGLD) have been extensively studied. Several theoretical frameworks have been presented to study this problem from different perspectives, such as information theory and stability. In this paper, we present a unified view from privacy leakage analysis to investigate the generalization bounds of SGLD, along with a theoretical framework for re-deriving previous results in a succinct manner. Aside from theoretical findings, we conduct various numerical studies to empirically assess the information leakage issue of SGLD. Additionally, our theoretical and empirical results provide explanations for prior works that study the membership privacy of SGLD.