The Unseen Species Problem Revisited

TL;DR

This paper revisits the unseen species problem, proposing new estimators and prediction intervals for different sample sizes, with theoretical guarantees and improved empirical performance.

Contribution

It introduces a new estimator for intermediate sample sizes, constructs principled prediction intervals, and extends guarantees to incidence data without independence assumptions.

Findings

The Good-Toulmin estimator is unique for small m and respects problem symmetries.

A new estimator significantly improves worst-case MSE for intermediate m.

For large m, a simple estimator matches the rate and outperforms recent methods.

Abstract

Given $n$ i.i.d. samples from an unknown discrete distribution over an unknown set, the unseen species problem is to predict how many new outcomes would be observed in $m$ additional samples. For small $m$ we show that the Good-Toulmin estimator is the unique estimator which both respects the symmetries of the problem and has non-trivial rate. We resolve the open problem of constructing principled prediction intervals for it. For intermediate $m$ we propose a new estimator which has a vastly improved worst case MSE compared to competing methods and we expect that our method can be applied to other species sampling problems. For large $m$ we follow previous authors in assuming a power law tail and show that a simple estimator achieves the same rate and better empirical performance than a recent sophisticated method. Moreover, we give pre-asymptotic guarantees. We extend the rate guarantees to incidence data, without further independence assumptions, provided that the sets are of bounded size. In the process we use Stein's method to obtain concentration inequalities for some natural functionals of sequences of i.i.d. discrete-set-valued random variables which are of independent interest.