Exact Good-Turing characterization of the two-parameter Poisson-Dirichlet superpopulation model

TL;DR

This paper proves that for finite samples, Bayesian nonparametric estimation of discovery probabilities under the two-parameter Poisson-Dirichlet model is exactly equivalent to the Good-Turing estimator, extending previous asymptotic results.

Contribution

It establishes an exact finite-sample equivalence between Good-Turing and Bayesian nonparametric estimators under the two-parameter Poisson-Dirichlet model, improving prior asymptotic results.

Findings

Exact Good-Turing estimation for finite samples.

Bayesian nonparametric estimation aligns with Good-Turing under superpopulation assumptions.

Interpretation of Good-Turing as a Bayesian estimator with partial information.

Abstract

Large sample size equivalence between the celebrated {\it approximated} Good-Turing estimator of the probability to discover a species already observed a certain number of times (Good, 1953) and the modern Bayesian nonparametric counterpart has been recently established by virtue of a particular smoothing rule based on the two-parameter Poisson-Dirichlet model. Here we improve on this result showing that, for any finite sample size, when the population frequencies are assumed to be selected from a superpopulation with two-parameter Poisson-Dirichlet distribution, then Bayesian nonparametric estimation of the discovery probabilities corresponds to Good-Turing {\it exact} estimation. Moreover under general superpopulation hypothesis the Good-Turing solution admits an interpretation as a modern Bayesian nonparametric estimator under partial information.

Equations

$$D_n^{AGT}(l)\approx \frac{l+1}{n}{c}_{l+1,n}$$

$$c_{l,n}^{\prime }=\frac{\alpha (1-\alpha {)}_{l-1}}{l!}{k}_n,$$

$$D_n^{BNP}(l)=D_n^{AGT}(l)\approx \frac{l+1}{n}{c}_{l+1,n}^{\prime }\approx \frac{l+1}{n}\frac{\alpha (1-\alpha {)}_l{k}_n}{(l+1)!}=\frac{k_n}{n}\frac{\alpha (1-\alpha {)}_l}{l!}.$$

$$E_N[K_n(p)]=E_N[\sum _{j=1}^{s}1\{N_{j,n}>0\}]=\sum _{j=1}^s[1-(1-p_j{)}^n]=s-\sum _{j=1}^s(1-p_j{)}^n,$$

$$E_N[C_{l,n}(p)]=E_N[\sum _{j=1}^{s}1\{N_{j,n}=l\}]=\sum _{j=1}^s\left(\genfrac{}{}{0ex}{}{n}{l}\right)p_j^l(1-p_j{)}^{n-l}.$$

$$E_P[\sum _{j=1}^{\infty }f(P_j)]=E_{{\tilde{P}}_1}[{\tilde{P}}_1^{-1}f({\tilde{P}}_1)].$$

$$E_{N,P}[K_n(P))]$$

$$E_{N,P}[C_{l,n}(P)]$$

$$p(n_1,\dots ,n_k)=\sum _{(i_1,\dots ,i_k)}E\left[\prod _{j=1}^k{P}_{i_j}^{n_j}\right],$$

$$p_{\alpha ,V}(n_1,\dots ,n_k)=V_{n,k}\prod _{j=1}^k(1-\alpha {)}_{n_j-1},$$

$$V_{n,k}=(n-k\alpha )V_{n+1,k}+V_{n+1,k+1},$$

$${\mathbb{E}}_{\alpha ,V}\left[(C_{l,n}{)}_{[r]}\right]=\frac{n![(1-\alpha {)}_{l-1}{]}^r}{(l!{)}^r(n-lr)!}\sum _{k-r=0}^{n-rl}{V}_{n,k}{S}_{n-rl,k-r}^{-1,-\alpha },$$

$$E_{\alpha ,V}[C_{l,n}(P)]=\left(\genfrac{}{}{0ex}{}{n}{l}\right)(1-\alpha {)}_{l-1}\sum _{k-1=0}^{n-l}{V}_{n,k}{S}_{n-l,k-1}^{-1,-\alpha }$$

$$E_{\alpha ,V}[K_n(P)]$$

$$Pr(Q_l=p_{\mu }|H)=Pr\{{\tilde{P}}_1=p_{\mu }|N_{1,n}=l,H\}=\frac{p_{\mu }^l(1-p_{\mu }{)}^{n-l}}{{\sum }_{j=1}^s{p}_j^l(1-p_j{)}^{n-l}}.$$

$$E_p[{\tilde{P}}_1|N_{1,n}=l,H]=\frac{{\sum }_{j=1}^s{p}_j^{l+1}(1-p_j{)}^{n-l}}{{\sum }_{j=1}^s{p}_j^l(1-p_j{)}^{n-l}}=\frac{(l+1)}{(n+1)}\frac{E_{N+1}(C_{l+1,n+1}(p)|H)}{E_N(C_{l,n}(p)|H)},$$

$$E_p[({\tilde{P}}_1)|N_{1,n}=l]\approx \frac{l+1}{n}\frac{c_{l+1,n}}{c_{l,n}}$$

$$E_P[{\tilde{P}}_1|N_{1,n}=l]$$

$$=$$

Full text

1 Introduction

In 1953 I. J. Good published the results of his collaboration with Alan Turing and their efforts at Bletchley Park to crack German ciphers for the Enigma machine during the World War II. The main result of the celebrated paper The population frequencies of species and the estimation of population parameters is the approximated Good-Turing estimator

[TABLE]

of the probability to observe a species already observed $l$ times in a sample of size $n$ assuming virtually nothing about the underlying population, for $c_{l+1,n}$ the number of species seen $l+1$ times. Good’s purpose in writing the paper was to provide a rigorous mathematical setting where to place Alan Turing’s first intuition. He eventually shows that (1) is nothing but an approximated version of the exact Bayesian estimator which could be calculated if the population frequencies were known and that even more accurate approximations may be obtained by suitably smoothing the unknown counting frequencies (cfr Section 8, Good 1953).

In 2016 Favaro et al. establish an asymptotical equivalence, between the modern Bayesian nonparametric estimator and the celebrated Good-Turing estimator. They demonstrate that, under two-parameter Poisson-Dirichlet $(\alpha,\theta)$ priors (Pitman and Yor, 1997) on the unknown species abundances, the posterior expected probability to discover a species observed $l$ times in a sample of size $n$ agrees, as $n$ tends to infinity, with the approximated Good-Turing estimator (1) if the frequency counts were smoothed by

[TABLE]

for $\alpha\in(0,1)$ and $k_{n}$ the number of different species observed, i.e.

[TABLE]

The purpose of this paper is to improve on this result by showing that a more in depth analysis of Good’s construction allows to prove that, for any finite sample size, under the same prior hypothesis, the Bayesian nonparametric estimator of the discovery probability corresponds to exact Good-Turing estimator (Good, 1953). Moreover, by means of a suitably reformulation of Good’s experimental conditions, we point out that, under general prior superpopulation on the unknown species abundances, the exact Good-Turing solution admits an interpretation as a modern Bayesian nonparametric solution under partial information. In other words it turns out that the two estimators just differ for the amount of information from the sample they assume to be available to the experimenter.

Let $N=(N_{1,n},\dots,N_{s,n})$, for $N_{j,n}\geq 0$ be the random abundances vector for a sample of size $n$ taken with replacement from a multinomial model of fixed parameters (known species relative abundances) $p=(p_{1},\dots,p_{s})$. Then $N_{j,n}\sim Bin(n,p_{j})$ and $P(N_{j}>0)=1-(1-p_{j})^{n}$. Let $K_{n}=\sum_{j=1}^{s}1\{N_{j,n}>0\}$ be the random number of different species in the sample and $(C_{1,n},\dots,C_{l,n},\dots,C_{n,n})$ the random vector of the number of species represented $1,\dots,l,\dots,n$ times, then

[TABLE]

and

[TABLE]

Now assume that $P=(P_{j})_{j\geq 1}$ is an infinite random discrete distribution, such that $\sum_{j}P_{j}=1$, let $(\tilde{P}_{j})_{j\geq 1}$ be its sized-biased permutation and recall that, for $\tilde{P}_{1}$ the random relative abundance of the first size-biased pick from $P$ and $f$ an arbitrary non-negative measurable function, then

[TABLE]

Let $N_{1},\dots,N_{K_{n}}$, for $N_{j}>0$, be the random abundances vector of random size $K_{n}$ of the different picks in order of appearance observed in a sample of size $n$ taken from $P$ then the analogues of (4) and (5) yield

[TABLE]

and

[TABLE]

1.1 Gibbs-type superpopulations

Gibbs-type models (Gnedin and Pitman, 2006) are a popular class of species sampling models generalizing the random discrete distribution characterizing the random weights of the infinite sum representation of the Dirichlet process (Pitman, 1996, 2003, 2006), while preserving the product form of the induced exchangeable partition probability function

[TABLE]

where $n_{i}=|A_{i}|$, $(i_{1},\dots,i_{k})$ ranges over all ordered $k$-tuples of distinct positive integers and $(P_{i})_{i\geq 1}$ is any rearrangements of the ranked atoms $(P_{i}^{\downarrow})_{i\geq 1}$ of $P$. Those models are characterized by having (9) in the product form

[TABLE]

for $\alpha\in(-\infty,1)$ and $V=(V_{n,k})$ weights satisfying the backward recursive relation

[TABLE]

where $V_{1,1}=1$ and $(x)_{y}=(x)(x+1)\cdots(x+y-1)$ is the usual notation for rising factorials. Given $(n_{1},\dots,n_{k})$ the multiplicities of the first $k$ species observed in a random sample of size $n=\sum_{i}n_{i}$, the probabilities to observe the $j$th old species or a new yet unobserved species at step $n+1$ are well-known to correspond respectively to $p_{\alpha,V}(n_{j}^{+})={V_{n+1,k}}(n_{j}-\alpha)/{V_{n,k}}$ and $p_{\alpha,V}(n^{k+1})=V_{n+1,k+1}/V_{n,k}$.

The $r$-th falling factorial moment of $C_{l,n}$ under a general Gibbs-type model $(\alpha,V)$ has been derived in Cerquetti (2013) (see also Favaro et al. 2013) and is given by

[TABLE]

where $S_{n,k}^{-1,-\alpha}$ stands for the generalized Stirling number of parameters $(-1,-\alpha)$. For $r=1$,

[TABLE]

and the prior expected number of different species in a sample of size $n$ specializes as

[TABLE]

Notice that both (13) and (1.1) may be derived directly from (1) and (1) if the law of $\tilde{P}_{1}$, the structural distribution of the prior $P=(P_{i})_{\geq 1}$ and its expected value are available in closed form.

2 Reformulating exact Good-Turing estimation

Exact Good-Turing estimation of the probability to discover a species encountered $l$ times in a sample of size $n$ is introduced in Good (1953) as follows:

*Suppose that a species is selected at random or rather equiprobably from an infinite population of $s$ species with fixed unknown population frequencies $p_{1},\dots,p_{s}$ and that then its number of occurrences in a sample of size $n$ is $l$. Let $H$ be the statistical hypothesis on $(p_{1},\dots,p_{s})$. Thus the initial (prior) probability that the species is the $j$-th of population frequency $p_{j}$ is $1/s$ and the likelihood that the observed number of occurrences is $l$ is ${n\choose l}p_{j}^{l}(1-p_{j})^{n-l}$. *

Notice that setting a uniform prior $1/s$ for the probability of every single species and then conditioning to the event that the species $j$ has been observed $l$ times corresponds to setting a prior $(p_{1},\dots,p_{s})$ of the probability of the single species and conditioning to $l-1$ observations of the same species $j$ observed in the first sampled individual. It follows that Good’s experimental setting can be equivalently reformulated as follows:

Assume to have a infinite population $p$ of $s$ species, with fixed unknown population frequencies $p_{1},\dots,p_{s}$. Without loss of generality let $\tilde{P}_{1}$ be the random population frequency of the first encountered species, taking values $(p_{1},\dots,p_{s})$ with prior probability respectively $(p_{1},\dots,p_{s})$. Now, consider the random variable $N_{1,n}-1$, i.e. the number of the observations of the first observed species, in a sample of additional $n-1$ observations. Then $N_{1,n}-1|\tilde{P}_{1}=p_{\mu}\sim Bin(n-1,p_{\mu})$ and the likelihood that the total number of occurrences of the first species in a random sample of size $n$ is $l$ is given by ${n-1\choose l-1}p_{\mu}^{l-1}(1-p_{\mu})^{n-l}$.

Then Good writes:

We shall consider the posterior expectation of $Q_{l}$, the unknown population frequencies of an arbitrary species that is represented $l$ times in the sample.

We can then rewrite Good’s exact result (cfr. eq. 12 in Good, 1953) for the posterior probability of the population frequency of a species encountered $l$ times in a sample of size $n$ as follows

[TABLE]

By an application of (5) the posterior expectation (cfr eq. 15 in Good, 1953) results

[TABLE]

for $C_{l,n}$ the random number of species represented $l$ times in a sample of size $n$.

Assuming virtually nothing about the underlying population $H(p_{1},\dots,p_{s})$, hence being impossible to evaluate (15), Good derives the estimator originally devised by Alan Turing observing that it makes sense to approximate (15) with

[TABLE]

for $c_{l,n}$ the number of species actually observed $l$ times in the $n$ sample. Then discusses the possibility to obtain even more accurate approximations by suitably smoothing the $c_{l,n}$’s (cfr Section 3, Good 1953).

Remark 1*.*

It is worth to notice at that point that in (15) Good is just conditioning on the observed frequency of the single species under consideration, nevertheless in (16) he assumes the whole vector of sample frequencies to be known, being impossible otherwise for $c_{l,n}$ and $c_{l+1,n}$ to be available. We will see in what follows that this discrepancy about the information available to the experimenter is what differentiates Good’s approach from the modern bayesian approach and how it vanishes under the two-parameter Poisson-Dirichlet superpopulation hypothesis.

3 Main results

Few lines later his main result Good (1953) clarifies:

If a specific (statistical) hypothesis $H(p_{1},\dots,p_{s})$ is accepted it is clearly not necessary to use the approximations since equation (15) can then be used directly. Similarly, if $H$ is assumed to be selected from a superpopulation, with an assigned probability density, then again it is theoretically possible to dispense with the approximation.

Let suppose that $H(p)$ is assumed to be selected from a superpopulation $H(P)$ for $P=(P_{j})_{j\geq 1}$ a random discrete distribution model for the unknown population frequencies, that here, without loss of generality, we assume to be countable infinite. Then an application of (15) yields

[TABLE]

where the last equivalence follows from (1). An application of (17) and (13) provides the following result