Sharp regret-Hellinger bounds for Gaussian empirical Bayes via polynomial approximation

TL;DR

This paper introduces a polynomial approximation technique to derive sharper, often optimal, regret bounds in Gaussian empirical Bayes, improving upon classical methods especially for compactly supported priors.

Contribution

The paper presents a new polynomial approximation approach that directly bounds unregularized regret, simplifying analysis and achieving sharper bounds in Gaussian empirical Bayes.

Findings

Sharp regret bound of $O(rac{ ext{distance}^2 imes ext{log}(1/ ext{distance})}{ ext{loglog}(1/ ext{distance})})$ for compact priors.

Method extends to priors with exponential tails, providing broader applicability.

Demonstrates regularization necessity for heavy-tailed priors under bounded moment assumptions.

Abstract

A central problem in the theory of empirical Bayes is to control the regret (excess risk) of a learned Bayes rule by the Hellinger distance between the estimated and true marginal densities. In the normal means model, the classical result of Jiang and Zhang (2009, Annals of Statistics) achieves this only after regularizing the Bayes rule and incurs an extraneous cubic logarithmic factor through a delicate recursive argument. This paper introduces a new technique, based on polynomial approximation and Bernstein-type inequalities for weighted $L_2$ norms, that bounds the unregularized regret directly. The method is conceptually simpler and yields sharper, sometimes optimal, regret bounds. For compactly supported priors, we prove the sharp bound that the regret is at most $O(\epsilon^2 \log(1/\epsilon)/\log\log(1/\epsilon))$, where $\epsilon$ is the Hellinger distance between the marginal densities. The same method also extends to priors with exponential tails. Conversely, we show that regularization is genuinely necessary for heavy-tailed priors under only bounded moment assumptions. As a statistical consequence, we obtain improved regret bounds for the nonparametric maximum likelihood estimator.