The Harmonic Entropy Estimator: Minimax Optimality and Semiparametric Efficiency for Infinite Alphabets

TL;DR

This paper introduces the harmonic entropy estimator for infinite discrete distributions, achieving minimax optimality and semiparametric efficiency under certain tail decay conditions, with strong theoretical guarantees.

Contribution

It proposes a novel harmonic entropy estimator that attains optimal convergence rates and efficiency for infinite alphabets, extending entropy estimation theory.

Findings

Achieves $1/n$ convergence rate for distributions with tail decay $p_j ot o 0$ too slowly.

Proves semiparametric efficiency under faster tail decay $p_j = o(j^{-2})$.

Unifies finite and infinite support entropy estimation theory.

Abstract

This paper considers the estimation of Shannon entropy for discrete distributions with countably infinite support. While minimax rates for finite-support distributions are established, infinite-support distributions present distinct challenges regarding bias control as probabilities vanish. We address this by introducing the \textit{harmonic entropy estimator}, a statistic derived from an exact algebraic identity relating the expectation of harmonic-transformed binomial counts to the logarithm of underlying success probabilities. We establish two main results characterizing the statistical limits of this problem. First, for the class of distributions with at least quadratically decaying tails ($p_j\lesssim j^{-2}$), we prove that the estimator achieves the parametric $L_2$-minimax convergence rate of order $1/n$. Second, under the stronger condition $p_j =o(j^{-2})$, we demonstrate that the estimator is semiparametrically efficient, converging to a normal distribution with variance matching the asymptotic efficiency bound $\textrm{Var}[\log p(X)]$. These results unify entropy estimation theory for finite-variance distributions, and provide a simple, one-step estimator with sharp theoretical guarantees.