$L^1$ Estimation: On the Optimality of Linear Estimators

TL;DR

This paper investigates the conditions under which linear estimators are optimal for $L^1$ and other $L^p$ loss functions, revealing that Gaussian priors uniquely induce linearity for $p eq 2$, with extensions to exponential family noise models.

Contribution

It establishes that Gaussian priors are the only ones inducing linear Bayesian estimators under $L^1$ and $L^p$ losses for $p eq 2$, and explores conditions for linearity in various noise models.

Findings

Gaussian prior uniquely induces linearity for $L^1$ loss.

Symmetric conditional distributions imply Gaussianity.

For $p eq 2$, only Gaussian priors induce linear estimators.

Abstract

Consider the problem of estimating a random variable $X$ from noisy observations $Y = X+ Z$, where $Z$ is standard normal, under the $L^1$ fidelity criterion. It is well known that the optimal Bayesian estimator in this setting is the conditional median. This work shows that the only prior distribution on $X$ that induces linearity in the conditional median is Gaussian. Along the way, several other results are presented. In particular, it is demonstrated that if the conditional distribution $P_{X|Y=y}$ is symmetric for all $y$, then $X$ must follow a Gaussian distribution. Additionally, we consider other $L^p$ losses and observe the following phenomenon: for $p \in [1,2]$, Gaussian is the only prior distribution that induces a linear optimal Bayesian estimator, and for $p \in (2,\infty)$, infinitely many prior distributions on $X$ can induce linearity. Finally, extensions are provided to encompass noise models leading to conditional distributions from certain exponential families.