Minimax Rate-Optimal Estimation of Divergences between Discrete Distributions

TL;DR

This paper introduces a new minimax rate-optimal estimator for alpha-divergences between discrete distributions that avoids common simplifying techniques, using a hybrid approach with linear programming and bias correction.

Contribution

It presents the first estimator that is minimax rate-optimal without relying on Poissonization, sample splitting, or polynomial approximation methods.

Findings

Achieves minimax rate-optimal estimation of alpha-divergences.

Does not require Poissonization or sample splitting.

Combines linear programming with bias-corrected plug-in estimators.

Abstract

We study the minimax estimation of $\alpha$-divergences between discrete distributions for integer $\alpha\ge 1$, which include the Kullback--Leibler divergence and the $\chi^2$-divergences as special examples. Dropping the usual theoretical tricks to acquire independence, we construct the first minimax rate-optimal estimator which does not require any Poissonization, sample splitting, or explicit construction of approximating polynomials. The estimator uses a hybrid approach which solves a problem-independent linear program based on moment matching in the non-smooth regime, and applies a problem-dependent bias-corrected plug-in estimator in the smooth regime, with a soft decision boundary between these regimes.