An Empirical Bayes Perspective on Heteroskedastic Mean Estimation

TL;DR

This paper develops an empirical Bayes method for heteroskedastic mean estimation with unknown variances, achieving near-optimal error bounds and adaptively handling the subset-of-signals problem without tuning.

Contribution

It introduces a profile MLE approach modeling observations as normal scale mixtures, with a sharper entropy bound and adaptive minimax rate achievement in heteroskedastic settings.

Findings

Achieves near-optimal mean squared error bounds.

Adaptively attains minimax rates in subset-of-signals problems.

Provides a sharper entropy bound for normal scale mixtures.

Abstract

Towards understanding the fundamental limits of estimation from data of varied quality, we study the problem of estimating a mean parameter from heteroskedastic Gaussian observations where the variances are unknown and may vary arbitrarily across observations. While a simple linear estimator with known variances attains the smallest mean squared error, estimation without this knowledge is challenging due to the large number of nuisance parameters. We propose a simple and principled approach based on empirical Bayes: model the observations as if they were i.i.d. from a normal scale mixture and compute the profile maximum likelihood estimator (MLE) for the mean, treating the nonparametric mixing distribution as nuisance. Our result shows that this estimator achieves near-optimal error bounds across various heteroskedastic models in the literature. In particular, for the subset-of-signals problem where an unknown subset of observations has small variance, our estimator adaptively achieves the minimax rate for all signal sizes, including the sharp phase transition, without any tuning parameters. One of our key technical steps is a sharper metric entropy bound for normal scale mixtures, obtained via Chebyshev approximations on a transformed polynomial basis. This approach yields an improved polylogarithmic, rather than polynomial, dependence on the variance ratio, which could be of independent interest.