Estimation of Entropy in Constant Space with Improved Sample Complexity

TL;DR

This paper introduces a new constant-memory method for estimating distribution entropy with improved sample complexity, reducing the number of samples needed while maintaining minimal memory usage.

Contribution

The authors present a novel entropy estimation algorithm that lowers sample complexity in constant space, improving upon previous methods.

Findings

Reduces sample complexity from (k/ε^3) to (k/ε^2) times polylog factors.

Maintains constant memory usage during estimation.

Conjectures optimality of the new sample complexity bound.

Abstract

Recent work of Acharya et al. (NeurIPS 2019) showed how to estimate the entropy of a distribution $\mathcal D$ over an alphabet of size $k$ up to $\pm\epsilon$ additive error by streaming over $(k/\epsilon^3) \cdot \text{polylog}(1/\epsilon)$ i.i.d. samples and using only $O(1)$ words of memory. In this work, we give a new constant memory scheme that reduces the sample complexity to $(k/\epsilon^2)\cdot \text{polylog}(1/\epsilon)$. We conjecture that this is optimal up to $\text{polylog}(1/\epsilon)$ factors.