A Stein Identity for q-Gaussians with Bounded Support

TL;DR

This paper introduces a new Stein identity for bounded-support q-Gaussians, enabling easier gradient estimation with reduced variance, which benefits machine learning tasks involving non-Gaussian distributions.

Contribution

The authors extend Stein's identity to bounded-support q-Gaussians, providing simplified gradient estimators and variance reduction techniques for non-Gaussian distributions.

Findings

Gradient estimators for q-Gaussians have low variance.

Bounded-support distributions improve gradient estimation in practice.

The method is applicable to Bayesian deep learning and sharpness-aware minimization.

Abstract

Stein's identity is a fundamental tool in machine learning with applications in generative models, stochastic optimization, and other problems involving gradients of expectations under Gaussian distributions. Less attention has been paid to problems with non-Gaussian expectations. Here, we consider the class of bounded-support $q$-Gaussians and derive a new Stein identity leading to gradient estimators which have nearly identical forms to the Gaussian ones, and which are similarly easy to implement. We do this by extending the previous results of Landsman, Vanduffel, and Yao (2013) to prove new Bonnet- and Price-type theorems for q-Gaussians. We also simplify their forms by using escort distributions. Our experiments show that bounded-support distributions can reduce the variance of gradient estimators, which can potentially be useful for Bayesian deep learning and sharpness-aware minimization. Overall, our work simplifies the application of Stein's identity for an important class of non-Gaussian distributions.