Extrapolating the profile of a finite population

TL;DR

This paper demonstrates that it is possible to consistently estimate the population profile, which is the distribution of types, from a small subsample in the sublinear regime, using a linear programming approach and complex analysis.

Contribution

It introduces a method to estimate the population profile in the sublinear sampling regime and characterizes its minimax optimality via an infinite-dimensional LP.

Findings

Consistent profile estimation is possible when sample size exceeds k / log k.

The optimal convergence rate in the linear regime is Theta(1 / log k).

A single infinite-dimensional LP characterizes the estimator's risk and optimality.

Abstract

We study a prototypical problem in empirical Bayes. Namely, consider a population consisting of $k$ individuals each belonging to one of $k$ types (some types can be empty). Without any structural restrictions, it is impossible to learn the composition of the full population having observed only a small (random) subsample of size $m = o(k)$. Nevertheless, we show that in the sublinear regime of $m =\omega(k/\log k)$, it is possible to consistently estimate in total variation the \emph{profile} of the population, defined as the empirical distribution of the sizes of each type, which determines many symmetric properties of the population. We also prove that in the linear regime of $m=c k$ for any constant $c$ the optimal rate is $\Theta(1/\log k)$. Our estimator is based on Wolfowitz's minimum distance method, which entails solving a linear program (LP) of size $k$. We show that there is a single infinite-dimensional LP whose value simultaneously characterizes the risk of the minimum distance estimator and certifies its minimax optimality. The sharp convergence rate is obtained by evaluating this LP using complex-analytic techniques.