Instance-Optimality for Private KL Distribution Estimation

TL;DR

This paper develops private algorithms for estimating unknown discrete distributions with optimality guarantees both in worst-case and local instance-specific scenarios, improving understanding of private KL divergence estimation.

Contribution

It introduces the first minimax optimal private estimators for KL divergence and proposes algorithms achieving instance-optimality in local neighborhoods, enhancing practical performance insights.

Findings

Proposed private estimators are minimax optimal for KL divergence.

Algorithms achieve instance-optimality up to constant factors.

Lower bounds are established using refined local neighborhoods.

Abstract

We study the fundamental problem of estimating an unknown discrete distribution $p$ over $d$ symbols, given $n$ i.i.d. samples from the distribution. We are interested in minimizing the KL divergence between the true distribution and the algorithm's estimate. We first construct minimax optimal private estimators. Minimax optimality however fails to shed light on an algorithm's performance on individual (non-worst-case) instances $p$ and simple minimax-optimal DP estimators can have poor empirical performance on real distributions. We then study this problem from an instance-optimality viewpoint, where the algorithm's error on $p$ is compared to the minimum achievable estimation error over a small local neighborhood of $p$. Under natural notions of local neighborhood, we propose algorithms that achieve instance-optimality up to constant factors, with and without a differential privacy constraint. Our upper bounds rely on (private) variants of the Good-Turing estimator. Our lower bounds use additive local neighborhoods that more precisely captures the hardness of distribution estimation in KL divergence, compared to ones considered in prior works.