MMSE Bounds Under Kullback-Leibler Divergence Constraints on the Joint Input-Output Distribution

TL;DR

This paper introduces tight bounds on MMSE under KL divergence constraints, useful for robust estimation and analysis in uncertain distribution scenarios, with practical applications demonstrated.

Contribution

It presents a novel family of MMSE bounds constrained by KL divergence, attained by Gaussian distributions, offering tighter and more general alternatives to classical bounds.

Findings

Bounds are tight and attained by Gaussian distributions.

Upper bound provides a minimax optimal estimator.

Applications demonstrate practical usefulness in signal processing.

Abstract

This paper proposes a new family of lower and upper bounds on the minimum mean squared error (MMSE). The key idea is to minimize/maximize the MMSE subject to the constraint that the joint distribution of the input-output statistics lies in a Kullback-Leibler divergence ball centered at some Gaussian reference distribution. Both bounds are tight and are attained by Gaussian distributions whose mean is identical to that of the reference distribution and whose covariance matrix is determined by a scalar parameter that can be obtained by finding the root of a monotonic function. The upper bound corresponds to a minimax optimal estimator and provides performance guarantees under distributional uncertainty. The lower bound provides an alternative to well-known inequalities in estimation theory, such as the Cram\'er-Rao bound, that is potentially tighter and defined for a larger class of distributions. Examples of applications in signal processing and information theory illustrate the usefulness of the proposed bounds in practice.