The variation of the posterior variance and Bayesian sample size determination

TL;DR

This paper investigates Bayesian sample size determination using the first two moments of the expected posterior variance, showing that a new criterion improves success rates and provides asymptotic insights, with applications to real datasets.

Contribution

It introduces a novel Bayesian sample size criterion based on the first two moments of the expected posterior variance, enhancing success rates over traditional methods.

Findings

Proposed criterion increases required sample size.

Significantly improves success rate for bounding posterior variance.

Provides asymptotic expressions and phase transition insights.

Abstract

We consider Bayesian sample size determination using a criterion that utilizes the first two moments of the expected posterior variance. We study the resulting sample size in dependence on the chosen prior and explore the success rate for bounding the posterior variance below a prescribed limit under the true sampling distribution. Compared with sample size determination based on the expected average of the posterior variance the proposed criterion leads to an increase in sample size and significantly improved success rates. Generic asymptotic properties are proven, such as an asymptotic expression for the sample size and a sort of phase transition. Our study is illustrated using two real world datasets with Poisson and normally distributed data. Based on our results some recommendations are given.

Equations

$$\pi (\theta |{\mathbf{x}}_n)\propto \pi (\theta )\cdot p({\mathbf{x}}_n|\theta ),$$

$$u_n^2:={\mathrm{Var}}_{\theta \sim \pi (\theta |{\mathbf{x}}_n)}(\theta ).$$

$$\hat{\theta }\pm u_n,$$

$$u_n^2<{\epsilon }^2$$

$$\overline{u_n^2}={\mathbb{E}}_{{\mathbf{x}}_n\sim m({\mathbf{x}}_n)}[u_n^2]<{\epsilon }^2,$$

$$\overline{u_n^2}+k\cdot \Delta u_n^2<{\epsilon }^2.$$

$$\Delta u_n^2=({\mathrm{Var}}_{x\sim m({\mathbf{x}}_n)}(u_n^2){)}^{1/2}.$$

$$p({\mathbf{x}}_n|\theta )=\prod _{i=1}^n\mathrm{Poi}(x_i|\theta )=\prod _{i=1}^n\frac{{\theta }^{x_i}}{x_i!}{e}^{-\theta },$$

$$\overline{u_n^2}={\mathbb{E}}_{{\mathbf{x}}_n\sim m({\mathbf{x}}_n)}[u_n^2]=\frac{\alpha }{\beta }\cdot \frac{1}{n+\beta }<{\epsilon }^2.$$

$$\overline{u_n^2}+k\cdot \Delta u_n^2<{\epsilon }^2,$$

$$\pi (\mu ,{\sigma }^2)=\mathcal{N}(\mu |{\mu }_0,\lambda {\sigma }^2)\cdot \mathrm{IG}({\sigma }^2|\alpha ,\beta ),$$

$$u_n^2={\mathrm{Var}}_{\mu \sim \pi (\mu |{\mathbf{x}}_n)}(\mu )=\frac{2\beta +{\sum }_{i=1}^n(x_i-\overline{x}{)}^2+\frac{n}{\lambda n_{\lambda }}(\overline{x}-{\mu }_0{)}^2}{n_{\lambda }(n+2\alpha -2)}.$$

$$\overline{u_n^2}+k\cdot \Delta u_n^2<{\epsilon }^2,$$

$$\theta =({\theta }_0,{\theta }^{\prime }).$$

$$\overline{u_n^2}+k\Delta u_n^2<{\epsilon }^2.$$

$$\begin{array}{rl}& \underset{n\to \infty }{\lim }{\mathbb{E}}_{{\mathbf{x}}_n\sim p({\mathbf{x}}_n|{\theta }_{\mathrm{true}})}[|n\cdot u_n^2-I_{{\theta }_{\mathrm{true},0}}^{-1}{|}^2]=0\text{mbox}and\\ & \underset{n\to \infty }{\lim }{\mathbb{E}}_{\theta \sim \pi (\theta )}[{\mathbb{E}}_{{\mathbf{x}}_n\sim p({\mathbf{x}}_n|\theta )}[|n\cdot u_n^2-I_{{\theta }_0}^{-1}{|}^2]]=0.\end{array}$$

$$\gamma =\frac{({\mathrm{Var}}_{\theta \sim \pi (\theta )}(I_{{\theta }_0}^{-1}){)}^{1/2}}{{\mathbb{E}}_{\theta \sim \pi (\theta )}[I_{{\theta }_0}^{-1}]}.$$

$$c_n=\frac{\Delta u_n^2}{\overline{u_n^2}}\text{mbox}and{c}_{\mathrm{true},n}=\frac{({\mathrm{Var}}_{{\mathbf{x}}_n\sim p({\mathbf{x}}_n|{\theta }_{\mathrm{true}})}(u_n^2){)}^{1/2}}{{\mathbb{E}}_{{\mathbf{x}}_n\sim p({\mathbf{x}}_n|{\theta }_{\mathrm{true}})}[u_n^2]}.$$

$$\underset{n\to \infty }{\lim }{c}_n=\gamma \text{mbox}and\underset{n\to \infty }{\lim }{c}_{\mathrm{true},n}=0.$$

$${\mathrm{Var}}_{{\mathbf{x}}_n\sim m({\mathbf{x}}_n)}(X_n)$$

$$\to {\mathrm{Var}}_{(\theta ,{\mathbf{x}}_n)\sim p(\theta ,{\mathbf{x}}_n)}(I_{{\theta }_0}^{-1})={\mathrm{Var}}_{\theta \sim \pi (\theta )}(I_{{\theta }_0}^{-1})$$

$$c_n\stackrel{X_n=n{u}_n^2}{=}\frac{({\mathrm{Var}}_{{\mathbf{x}}_n\sim m({\mathbf{x}}_n)}(X_n){)}^{1/2}}{{\mathbb{E}}_{{\mathbf{x}}_n\sim m({\mathbf{x}}_n)}[X_n]}\to \frac{({\mathrm{Var}}_{\theta \sim \pi (\theta )}(I_{{\theta }_0}^{-1}){)}^{1/2}}{{\mathbb{E}}_{\theta \sim \pi (\theta )}[I_{{\theta }_0}^{-1}]}=\gamma .$$

$$c_{\mathrm{true},n}=\frac{({\mathrm{Var}}_{{\mathbf{x}}_n\sim p({\mathbf{x}}_n|{\theta }_{\mathrm{true}})}(X_n){)}^{1/2}}{{\mathbb{E}}_{{\mathbf{x}}_n\sim p({\mathbf{x}}_n|{\theta }_{\mathrm{true}})}[X_n]}\to \frac{({\mathrm{Var}}_{{\mathbf{x}}_n\sim p({\mathbf{x}}_n|{\theta }_{\mathrm{true}})}(I_{{\theta }_{\mathrm{true},0}}^{-1}){)}^{1/2}}{{\mathbb{E}}_{{\mathbf{x}}_n\sim p({\mathbf{x}}_n|{\theta }_{\mathrm{true}})}[I_{{\theta }_{\mathrm{true},0}}^{-1}]}=0,$$

$$\underset{\epsilon \to 0}{\lim }\frac{n_{\epsilon }}{{\epsilon }^{-2}\cdot (1+k\gamma ){\mathbb{E}}_{\theta \sim \pi (\theta )}[I_{{\theta }_0}^{-1}]}=1$$

$$\begin{array}{rlr}{s}_n& :=n(\overline{u_n^2}+k\Delta u_n^2)=n\overline{u_n^2}(1+k{c}_n)& \stackrel{n\to \infty }{⟶}(1+k\gamma ){\mathbb{E}}_{\theta \sim \pi (\theta )}[I_{{\theta }_0}^{-1}]=:s_{\infty }.\end{array}$$

$$\underset{\epsilon \to 0}{\lim }{n}_{\epsilon }=\infty .$$

$$\underset{\epsilon \to 0}{\lim }\frac{n_{\epsilon }}{{\epsilon }^{-2}\cdot s_{\infty }}=1.$$

$$\displaystyle\liminf_{\varepsilon\rightarrow 0}\frac{n_{\varepsilon}}{\varepsilon^{-2}s_{\infty}}\overset{\mbox{\tiny def. of }n_{\varepsilon}}{\geq}\liminf_{\varepsilon\rightarrow 0}\frac{n_{\varepsilon}\cdot(\overline{u^{2}_{n_{\varepsilon}}}+k\,\Delta u^{2}_{n_{\varepsilon}})}{s_{\infty}}=\liminf_{\varepsilon\rightarrow 0}\frac{s_{n_{\varepsilon}}}{s_{\infty}}\overset{\mbox{\tiny\eqref{eq:s_n_convergence},\,\eqref{eq:n_eps_diverges}}}{=}1\,.$$

$$\underset{\epsilon \to 0}{\mathrm{limsup}}\frac{n_{\epsilon }}{{\epsilon }^{-2}\cdot s_{\infty }}\ge 1+2\delta .$$

$$1\le m_{\epsilon }<(1+\delta /2)(1+m_{\epsilon })\le n_{\epsilon }^{\prime }\le m_{\epsilon }(1+\delta )\stackrel{\text{mbox}\text{eqref}eq:absur{d}_{s}tatement}{<}{n}_{\epsilon }.$$

$$\overline{u_{n_{\epsilon }^{\prime }}^2}+k\cdot \Delta u_{n_{\epsilon }^{\prime }}^2=\frac{s_{n_{\epsilon }^{\prime }}}{n_{\epsilon }^{\prime }}\stackrel{n_{\epsilon }^{\prime }>m_{\epsilon }}{<}\frac{(1+\delta /2)\cdot s_{\infty }}{n_{\epsilon }^{\prime }}\stackrel{\text{eqref}eq:neste{d}_{i}neq}{\le }\frac{s_{\infty }}{1+\lfloor s_{\infty }{\epsilon }^{-2}\rfloor }\le {\epsilon }^2.$$

Full text

1 Introduction

Sample size determination (SSD) is the attempt to estimate the data size that is needed in order to meet a certain criterion [1]. This task is usually performed at a planning stage before any data is actually measured or recorded so that especially in the context of high financial or temporal expenses a careful SSD becomes indispensable. In the design, say, of animal experiments or clinical trials SSD can even have an ethical dimension [2, 3]. In this article we study a Bayesian method for SSD that limits the expected fluctuations of the uncertainty of the result. By “uncertainty” we will here mean (the square root of) the posterior variance. For $n$ data points $\mathbf{x}_{n}=(x_{1},\ldots,x_{n})$ drawn from a sampling distribution $p(\mathbf{x}_{n}|\theta)$ with parameter $\theta$ the posterior distribution is defined by

[TABLE]

where $\pi(\theta)$ denotes the prior for the parameter $\theta$. The posterior variance is then given as

[TABLE]

In practice, a scientist performing an experiment might desire to specify her/his result with an according uncertainty, say

[TABLE]

with $u_{n}$ being the square root of $u_{n}^{2}$ as defined in (2) and with $\hat{\theta}$ being the posterior mean. In order for this result to be precise enough the scientist might desire to fulfill a condition such as $u_{n}<\varepsilon$ or, equivalently,

[TABLE]

for some small, positive $\varepsilon$ that is chosen a priori. As the posterior distribution is dependent on the data $\mathbf{x}_{n}$, so is $u_{n}^{2}$. Choosing an appropriate sample size $n$ so that (3) is guaranteed before $\mathbf{x}_{n}$ is known is only possible for a few restricted scenarios, for instance Bernoulli distributed samples [4, 5]. A more generally applicable criterion is to require instead of (3)

[TABLE]

where $m(\mathbf{x}_{n})=\int p(\mathbf{x}_{n}|\theta)\pi(\theta)\mathrm{d}\theta$ denotes the prior predictive. This is known as the average posterior variance criterion (APVC) in the literature [6, 4, 7]. For many standard cases explicit expressions for $\overline{u_{n}^{2}}$ can be derived. The usage of the prior predictive $m(\mathbf{x}_{n})$ is quite natural as it describes what is known about the data $\mathbf{x}_{n}$ given our prior knowledge. We will denote the smallest $n$ such that the APVC (4) is satisfied throughout this article by $\widetilde{n}_{\varepsilon}$. In the literature many alternative criteria can be found that replace $u^{2}_{n}$ by some other, data dependent random variable $T(\mathbf{x}_{n})$, compare for instance [8, 9, 10, 6, 11]. While we will stick in this article to the choice $T(\mathbf{x}_{n})=u^{2}_{n}$ many of the ideas presented here can, in principle, be translated to such approaches. 111Section 3 of this article, for instance, relies on Assumption 3.1. Provided similar assumptions hold for $T(\mathbf{x}_{n})$ then all the proofs given there carry through.

The APVC has a rather obvious drawback: it only guarantees (3) to hold on average. Consequently this might lead to an uncertainty $u_{\widetilde{n}_{\varepsilon}}$ that is well beyond $\varepsilon$ for certain data samples $\mathbf{x}_{\widetilde{n}_{\varepsilon}}$, compare for instance Figure 4(a) below. To get a better grasp on the variability of the uncertainty one can make more extensive usage of the prior predictive $m(\mathbf{x}_{n})$ [10, 7, 12, 13, 14]. This article aims at studying the behavior of such criteria and thereby to give some guidance or, at least, a deeper understanding for a SSD based on $m(\mathbf{x}_{n})$. In order to do so we will use a extension of the APVC (4) which we will call the variation of the posterior variance criterion (VPVC) and which takes the form

[TABLE]

where $k$ is a parameter to be chosen and where

[TABLE]

Throughout this article we will denote the smallest $n$ such that (5) is satisfied by $n_{\varepsilon}$. We will see in Section 2 below that taking (5) into account leads to substantially different sample sizes than the sole consideration of the APVC (4) and to a better compliance with (3). We will often take $k=2$ in this article (loosely motivated from the normal distribution) but will show that for $\varepsilon\rightarrow 0$ there is an optimal $k_{\ast}$. We will give some ideas on how to guess $k_{\ast}$ in practice. Moreover, we will provide an asymptotic formula for the sample size $n_{\varepsilon}$ in the small $\varepsilon$ regime.

The VPVC (5) is of course not as exhaustive in its description of the variability of $u_{n}^{2}$ as the full law of $u_{n}^{2}$ under $m(\mathbf{x}_{n})$. In Figure 1 we can see however that for $k=2$ and the setup from Section 2.1 the left hand side of (5) roughly covers 95.8% of the law of $u_{n}^{2}$ under $m(\mathbf{x}_{n})$ and thus pretty accurately coincides with the intuition (for a normal random variable we would expect 97.7%). On the other hand, this simplification is quite convenient for our purposes: it provides us with explicit expressions, spares us numerical issues and allows for a rather concise discussion of asymptotic properties in Section 3. We will discuss our method for two common cases: for Poisson and normally distributed data and illustrate our discussion with real world datasets.

The authors are, to the best of their knowledge, not aware of work in the literature that considers a criterion in the exact same shape of (5). In [4] Pham-Gia and Turkkan consider an object such as $\Delta u_{n}^{2}$ from (6) for a Binomial distribution but apply it in a different manner. Our approach is inspired from the quite common idea of using $m(\mathbf{x}_{n})$ for studying $u_{n}^{2}$. We believe that the discussion in this article deepens the understanding of SSD methods build on $m(\mathbf{x}_{n})$ in general.

The article is organized as follows: Section 2 discusses the application of the VPVC to Poisson and normally distributed data. We will compare our results to the ones of the simpler AVPC method (4), visualize the effect of using a prior that is inconsistent with the underlying parameter and debate how a conservative SSD could be performed. For this purpose we will use actual datasets, namely the goals from international football matches in the years 2015-2019 [15] and the length of pop songs from the Million song dataset [16]. In Section 3 we will look at the behavior of the VPVC for $\varepsilon\rightarrow 0$. The results from that section, especially Theorem 3.5, indicate that SSD methods based on $m(\mathbf{x}_{n})$ exhibit some sort of phase transition in limit. We will show that a $k_{\ast}$ exists such that for $k>k_{\ast}$ a SSD based on the VPVC (5) will ensure that (3) will asymptotically be true with probability 1. As $k_{\ast}$ will depend on the (usually unknown) true value of $\theta$, we will discuss a method how to get an upper bound based on the prior knowledge. Moreover, in Lemma 3.4 we will provide a generic, asymptotic formula for the sample $n_{\varepsilon}$ predicted the VPVC.

2 SSD based on the variation of the posterior variance

In this section we study the effects of using the VPVC as proposed in (5) and discuss its dependency on the prior knowledge and the true parameter. The discussion is first carried out for the case of a single parameter and Poisson distributed data. In Section 2.2 we then consider an example that involves nuisance parameters and normally distributed data.

2.1 Single Parameter: Poisson distributed data

The histogram in Figure 2 shows the number of goals scored in 5784 international football matches that were played between 2014 and 2019, taken form [15]. Suppose we want to get an estimate of the average number of goals $\theta$ that are scored in a game (from both teams). Of course, if we have the full dataset depicted in Figure 2 we can simple take the sample mean, which yields in fact an average of $2.71$ goals per game in the mentioned time range.

If we are not in possession of the full dataset the question arises how many samples, that is football matches in this case, do we need before we can make a “decent” guess about $\theta$. To formalize this question let us suppose that the data for $n$ games follows the product of $n$ Poisson distributions with the (unknown) parameter of interest $\theta$: 222This is an approximation. The ratio of variance and mean is around 1.3 for the full dataset, however to keep the setup simple we will stick to the Poisson assumption and treat the mean, i.e. the MLE, of 2.71 as the true parameter.

[TABLE]

where $\mathbf{x}_{n}=(x_{1},\ldots,x_{n})$ and each $x_{i}$ for $i=1,\ldots,n$ should be read as the total number of goals in game $i$. How large should we choose $n$ to get an estimate for $\theta$? A somewhat natural idea is to require that $n$ should be large enough so that the uncertainty $u_{n}$ we can specify for the result is smaller than an $\varepsilon$, say $\varepsilon=0.3$. Recall however that we want to find $n$ before we measure any data, that is, in our example, before we know any scores. Usually, $u_{n}$ will depend on the available data, so how can we find $n$ before we know $\mathbf{x}_{n}$?

At this point Bayesian statistics comes in handy as it allows us to use prior knowledge for this purpose, condensed in a prior distribution $\pi(\theta)$. We will here use a gamma distribution $\pi(\theta)=\mathrm{Gamma}(\theta|\alpha,\beta)$ with shape parameter $\alpha$ and rate $\beta$. We can then compute the prior predictive $m(\mathbf{x}_{n})$ as in (4) which gives us a distribution for the data $\mathbf{x}_{n}$ given our prior knowledge. As uncertainty $u_{n}$ for our result we take the square root of the posterior variance $u_{n}^{2}=\frac{\alpha+\sum_{i=1}^{n}x_{i}}{(n+\beta)^{2}}$. The inequality $u_{n}^{2}<\varepsilon^{2}$ has no solution for $n$ that holds for any $\mathbf{x}_{n}$. However, we can require this inequality to hold on average over the available information $m(\mathbf{x}_{n})$ about $\mathbf{x}_{n}$. In other words: choose the smallest $n\geq 1$ such that

[TABLE]

We referred to this as the APVC in the introduction and denoted the corresponding $n$ by $\widetilde{n}_{\varepsilon}$. The result of such an SSD for $\varepsilon=0.3$ is shown in Figure 3(a): for various values of the prior mean $\mathbb{E}_{\theta\sim\pi(\theta)}[\theta]=\frac{\alpha}{\beta}$ and its standard deviation $((\mathrm{Var}_{\theta\sim\pi(\theta)}(\theta)))^{1/2}=\frac{\alpha^{1/2}}{\beta}$ we plotted the sample size $\widetilde{n}_{\varepsilon}$ predicted by (8). Naturally, as the mean of $\pi(\theta)$ increases the sample size $\widetilde{n}_{\varepsilon}$ increases as well since $\theta$ is linked to the variance of the Poisson distributed data. The standard deviation of $\pi(\theta)$ seems to have only a minor influence on the sample size with one exception: below a certain threshold the prior variance pushes the posterior variance into the right ballpark so that even a minimal sample size of $\widetilde{n}_{\varepsilon}=1$ is enough to fulfill (8).

Figure 4(a) illustrates for which prior choices the SSD based on the APVC is successful based on the football goal dataset we depicted in Figure 2: for each prior mean and standard deviation and the corresponding sample sizes $\widetilde{n}_{\varepsilon}$ from Figure 3(a) 1000 random samples of size $\widetilde{n}_{\varepsilon}$ were drawn from the football dataset and the corresponding $u_{\widetilde{n}_{\varepsilon}}^{2}$ were computed. Figure 4(a) shows the fraction of $u_{\widetilde{n}_{\varepsilon}}^{2}$ that is actually below $\varepsilon^{2}$. For comparison the “true” value (the average goal number in the full dataset shown in Figure 2) is depicted by the dashed line. The result is rather disappointing. Only beyond the true $\theta$ of $2.71$ the quota rises above $60\%$. The reason for this unpleasant effect is quite apparent: by construction, the APVC only requires $u_{\widetilde{n}_{\varepsilon}}^{2}$ to be small enough on average.

As we pointed out in the introduction we therefore here use a refined criterion, which we called the VPVC and which takes the form

[TABLE]

where $\Delta u_{n}^{2}=(\mathrm{Var}_{\mathbf{x}_{n}\sim m(\mathbf{x}_{n})}(u_{n}^{2}))^{1/2}=\frac{\alpha^{1/2}}{\beta}\cdot\frac{n^{1/2}}{(n+\beta)^{3/2}}$ and where we choose $k=2$. The result of the VPVC is depicted in Figure 3(b). There are two clear differences compared to Figure 3(a): first, the sample sizes $n_{\varepsilon}$ are higher than the numbers $\widetilde{n}_{\varepsilon}$ we obtained from the APVC, which was expected as we added an additional positive term $k\,\Delta u_{n}^{2}$ to the left hand side of the criterion. Second, and perhaps more important, the result of the SSD substantially increases once we increase the standard deviation of the prior. This allows us to make our sample choice more conservative for a given prior mean by increasing its standard deviation i.e our prior uncertainty about $\theta$, which is quite natural. The most conservative SSD is thus located in the right upper corner of the plot. This phenomenon for parameters that describe the variance of the data will appear again in the next subsection.

The success of the refined criterion is depicted in Figure 4(b). The percentage of $u_{n_{\varepsilon}}^{2}$ below $\varepsilon^{2}$ reaches now much higher values, at many positions beyond $95\%$. This success is also more robust against deviations from the true value. Even for a mean of the prior $\pi(\theta)$ which is well below the true $\theta$ a high enough standard deviation of $\pi(\theta)$ will allow to reach a quota of $95\%$.

Figure 4(b) therefore splits in three areas. As in Figure 4(a) there is a bottom part where the posterior variance is dominated by the small prior uncertainty so that $u_{n_{\varepsilon}}^{2}<\varepsilon^{2}$ is easily satisfied even for the minimal sample size of $n_{\varepsilon}=1$. As the standard deviation of $\pi(\theta)$ gets larger a too small value of the prior mean will result in a low quota: the prior was chosen too optimistic. Increasing either the mean of $\pi(\theta)$ or its standard deviation will lead however to a more conservative SSD and to a better compliance of $u_{n_{\varepsilon}}^{2}<\varepsilon^{2}$. The highest percentage can therefore be found in the upper right corner of Figure 4(b) and in the bottom area.

2.2 Nuisance parameters: normally distributed data

In this section we will consider a sample size determination for the identification of the “typical” song length $\mu$ of a pop song. For this purpose we will use the Million Song Dataset from [16] that contains metadata for a million pop songs, collected in 2011. Averaging over the length of all songs in the dataset reveals that $\mu$ is around 4.17 minutes. As before, our goal is however not to find an outmost precise value of $\mu$ but instead to identify a minimal number of songs $n$ from which we can estimate $\mu$ up to a pre-specified precision $\varepsilon$. For the values of $\varepsilon$ used in this article the sample size $n$ will turn out to be much smaller number than a million. This allows us, once more, to use the full dataset to judge the effectiveness of our SSD.

For the sake of simplicity we will make the assumption that song length is a property that’s normally distributed, thereby ignoring for instance possible skewness issues that arise from the fact that a song cannot have a negative length. For $n$ songs we assume that their lengths $\mathbf{x}_{n}=(x_{1},\ldots,x_{n})$ are normally distributed, i.e. $p(\mathbf{x}_{n}|\mu,\sigma^{2})=\prod_{i=1}^{n}\mathcal{N}(x_{i}|\mu,\sigma^{2})$ with mean $\mu$ and standard deviation $\sigma^{2}$. Both parameters $\mu$ and $\sigma^{2}$ are unknown. While $\mu$ is the parameter of interest, $\sigma^{2}$ will be considered as a nuisance parameter. We take the normal-inverse-gamma prior [17]

[TABLE]

where $\mathrm{IG}$ denotes the inverse-gamma distribution and where $\alpha>2,\lambda,\beta>0$ and $\mu_{0}$ are hyperparameters. The squared uncertainty $u_{n}^{2}$ for $\mu$ is given by the variance of the marginal posterior $\pi(\mu|\mathbf{x}_{n})$:

[TABLE]

where $n_{\lambda}=n+\lambda^{-1}$ and $\overline{x}=\frac{1}{n}\sum_{i=1}^{n}x_{i}$.

We will once more consider the VPVC, that is we choose the smallest $n=n_{\varepsilon}$ such that

[TABLE]

where $\overline{u_{n}^{2}}=\mathbb{E}_{\mathbf{x}_{n}\sim m(\mathbf{x}_{n})}[u_{n}^{2}]=\frac{\beta}{n_{\lambda}\,(\alpha-1)}$ and \Delta u_{n}^{2}=(\mathrm{Var}_{\mathbf{x}_{n}\sim m(\mathbf{x}_{n})}(u_{n}^{2}))^{1/2}=\allowbreak\frac{\beta}{n_{\lambda}(\alpha-1)(\alpha-2)^{1/2}}\big{(}\frac{n}{n+2\alpha-2}\big{)}^{1/2} and where we choose again $k=2$. The sample sizes $n_{\varepsilon}$ for various choices of the prior $\pi(\mu,\,\sigma^{2})$ and an $\varepsilon$ of $20\,\mathrm{sec}=0.33\,\mathrm{min}$ are depicted in Figure 5. To visualize the impact of the prior knowledge we varied one of the marginals $\pi(\mu)$ and $\pi(\sigma^{2})$ while keeping the other one fixed. In Figure 5(a) we varied the marginal prior $\pi(\mu)$, while fixing the marginal $\pi(\sigma^{2})$ to have a mean of $3.0\,\mathrm{min}^{2}$ and a standard deviation of $1.5\,\mathrm{min}^{2}$, for comparison this $\pi(\sigma^{2})$ was marked by the black cross in Figure 5(b). For Figure 5(b) we fixed $\pi(\mu)$ to have a mean $3.5\,\mathrm{min}$ and a standard deviation of $1.0\,\mathrm{min}$ (marked by the black cross in Figure 5(a)) and varied $\pi(\sigma^{2})$. In particular, the two crosses in Figure 5(a) and 5(b) both mark positions with an equal sample size of $n=49$. The APVC criterion from (4) and the average coverage criterion from [18, 19] both yield a sample size of 24 for this prior. Note, that the effect of the marginal $\pi(\mu)$ is rather minimal, once $\pi(\sigma^{2})$ is kept fixed. In fact, the VPVC turns out to be independent of the hyperparameter $\mu_{0}$ - compare the vertical symmetry in Figure 5(a). Varying the standard deviation of $\pi(\mu)$ while keeping $\pi(\sigma^{2})$ fixed will only affect the hyperparameter $\lambda$ which has only a minor influence on the SSD result. The only exception to this is the bottom area of Figure 5(a) where the small standard deviation of $\pi(\mu)$ forces the posterior variance to be small and thus predicts a small sample size of 1, which is similar to a phenomenon we observed in Section 2.1. Figure 5(b) shows that the variance parameter $\sigma^{2}$ has a similar influence on the sample size as the Poisson parameter in Section 2.1: increasing either the mean or the standard deviation of $\pi(\sigma^{2})$ will increase the sample size so that the most conservative experimental design is located in the upper right corner. This behavior can again be expected since increasing the mean of $\pi(\sigma^{2})$ one will expect data that has larger variability and contains less information. In contrast to Figure 3(b) and Figure 5(a) there is no distinct bottom area of minimal sample sizes.

As in Section 2.1 we can use the full dataset to judge the success of our sample size planning. Figure 6 displays, for the same priors as in Figure 5 the proportion of $u_{n_{\varepsilon}}^{2}$ below $\varepsilon^{2}$ for $u_{n_{\varepsilon}}^{2}$ computed from random samples of size $n_{\varepsilon}$, drawn from the Million Song Dataset. The left plot, Figure 6(a), reveals that the choice of the marginal $\pi(\mu)$ has mostly an impact on the success of the SSD if its mean is far-off from the true parameter. Moreover, a very precise prior knowledge about $\mu$, will result in a high proportion of $u_{n_{\varepsilon}}^{2}<\varepsilon^{2}$, which is similar to the observations we made for the Poisson example in Section 2.1. The behavior of the percentage for which $u_{n_{\varepsilon}}^{2}<\varepsilon^{2}$ when varying $\pi(\sigma^{2})$ is similar to what we have observed for the Poisson parameter in Section 2.1. For small values of the mean of $\pi(\sigma^{2})$ the quota drops, which can be cured by increasing the standard deviation of $\pi(\sigma^{2})$. The highest percentage can be found in the upper right corner. For the prior marked by the black crosses in Figure 6, the same as the one marked in Figure 5, the quota is around $88.7\%$ while the sample size predicted by the APVC and average coverage criterion only yields a percentage of around $49.9\%$. The bottom area of Figure 6(b), where the posterior variance is heavily influenced by the prior, is not as distinct as in Figure 4(b), this role seems to be played in this context by $\mu$ - compare Figure 6(a).

Let us shortly summarize the most important observations we have made concerning SSD based on the prior predictive $m(\mathbf{x}_{n})$ via using the VPVC (5). There are two strategies to have a good chance of achieving $u_{n}^{2}<\varepsilon^{2}$:

A precise prior knowledge about the parameter of interest. This will result in small sample sizes 2. 2.

A conservative estimate of the parameter that determines the variation of the data. This will result in larger sample sizes. We have observed two different ways to achieve this.

• •

Make a conservative (that is rather large) guess for the mean of the prior marginal w.r.t this parameter.

• •

Choose a large enough standard deviation for this marginal prior. In order for this to work it is not sufficient to base the SSD on the average $\overline{u_{n}^{2}}$ only but higher moments as in the VPVC have to be taken into account.

In the Poisson example from Section 2.1 both strategies 1 and 2 concern the same parameter $\theta$. For the case of normally distributed data we have observed that strategy 1 concerns the parameter of interest $\mu$ while to follow strategy 2 the parameter $\sigma^{2}$, that describes the variance of the data, was central.

3 Behavior for small $\varepsilon$

We have seen in Section 2 that the determination of the sample size via the VPVC can be successful, provided the prior knowledge is either precise enough or sufficiently conservative. Being “conservative” is of course a rather vague quality, but in this section we want to answer what happens if $\varepsilon$ from (5) becomes smaller. Will it become easier or harder to be conservative? In Theorem 3.5 we show that as $\varepsilon\rightarrow 0$ the VPVC tends to satisfy (3) perfectly provided $k$ is bigger then some threshold $k_{\ast}$. In other words for such a $k$ the VPVC has a tendency to become conservative. If $k$ is smaller than $k_{\ast}$ on the other hand we will show that, asymptotically, the uncertainty will almost surely be above the expression used for the VPVC. This phase transition with respect to $k$, compare Figure A.1 in the appendix, gives some inside on SSD based on the prior predictive in the small $\varepsilon$ regime. We will give some ideas how to get an upper bound on $k_{\ast}$. Another result of this section is Lemma 3.4, where we show an asymptotic formula for the sample size $n_{\varepsilon}$ determined by the VPVC.

First let us fix our setup. Similar as in Section 2 we will assume that there is univariate parameter of interest $\theta_{0}$ and a, possibly empty or multivariate, nuisance parameter $\theta^{\prime}$ so that the total parameter that determines our sampling distribution $p(\mathbf{x}_{n}|\theta)=p(x_{1},\ldots,x_{n}|\theta)$ is given by

[TABLE]

We will denote the Fisher information matrix of the single sample distribution $p(x_{1}|\theta)$ by $I_{\theta}$ and its first component (the one linked to $\theta_{0}$) by $I_{\theta_{0}}$. Note, that since $\theta_{0}$ is univariate $I_{\theta_{0}}$ is a non-negative number. The knowledge about $\theta$ is described by the prior $\pi(\theta)$ with marginals $\pi(\theta_{0})$ and $\pi(\theta^{\prime})$. We further fix a value $\theta_{\mathrm{true}}=(\theta_{\mathrm{true},0},\theta_{\mathrm{true}}^{\prime})$ in the domain of $\pi(\theta)$ that we treat as the true parameter. In practice $\theta_{\mathrm{true}}$ will of course be unknown. Given some data $\mathbf{x}_{n}=(x_{1},\ldots,x_{n})$ we introduce as above the squared uncertainty by $u_{n}^{2}=\mathrm{Var}_{\theta_{0}\sim\pi(\theta_{0}|\mathbf{x}_{n})}(\theta_{0})$ as well as $\overline{u_{n}^{2}}=\mathbb{E}_{\mathbf{x}_{n}\sim m(\mathbf{x}_{n})}[u_{n}^{2}]$, $\Delta u_{n}^{2}=(\mathrm{Var}_{\mathbf{x}_{n}\sim m(\mathbf{x}_{n})}(u_{n}^{2}))^{1/2}$ with $m(\mathbf{x}_{n})=\int p(\mathbf{x}_{n}|\theta)\pi(\theta)\mathrm{d}\theta$ and formulate the VPVC for $\varepsilon>0$ as

[TABLE]

The smallest $n$ satisfying (14) will be called once more $n_{\varepsilon}$.

Assumption 3.1**.**

We assume that for $\pi(\theta)$ and $\theta_{\mathrm{true}}$ it holds:

Fisher information sufficiently regular*: The Fisher information $I_{\theta_{0}}$ is strictly positive for $\theta=\theta_{\mathrm{true}}$ and almost every $\theta$ in the domain of $\pi(\theta)$. Moreover we assume that the second moment $\mathbb{E}_{\theta\sim\pi(\theta)}[I_{\theta_{0}}^{-2}]$ exists and that $\mathrm{Var}_{\theta\sim\pi(\theta)}(I_{\theta_{0}}^{-1})>0$.* 2. 2.

B.-v.-Mises limit holds in $L^{2}$**: The quantity $n\cdot u_{n}^{2}$ converges in $L^{2}$ against $I_{\theta_{0}}^{-1}$ conditional under $\theta=\theta_{\mathrm{true}}$ and $\pi(\theta)$ in the sense that:

[TABLE] 3. 3.

No trivial SSD*: For all $n$ we have $\overline{u_{n}^{2}}>0$.*

From these assumptions point 2 is probably the one that needs the most explanation. The Bernstein-von-Mises theorem [20] indicates that under relatively mild conditions, and conditional on $\theta$, the posterior distribution is asymptotically normal with variance $\frac{1}{n}I_{\theta_{0}}^{-1}$, which motivates (15). When considering variances, as in this article, it is then quite natural to require convergence in $L^{2}$, while the B.-v.-Mises theorem only ensures convergence in probability. While standard methods, such as Vitali’s convergence theorem, could be applied to provide the first convergence (15) these seems more involved for the second limit as it is in a rather non-standard form. Examining this point in more detail would stray far away from the scope of this article and could be subject to future research. However, we found that both identities (15) were rather easy to check for many standard cases such as the ones from Section 2 or a Bernoulli distribution with a Beta prior.

Under Assumptions 3.1 the posterior variance is, conditional on $\theta$, asymptotically proportional to $I_{\theta_{0}}^{-1}$. The following quantity therefore describes the variation of the asymptotic posterior variance given the prior knowledge $\pi(\theta)$:

[TABLE]

Let us make these considerations more precise.

Lemma 3.2**.**

Define the coefficient of variation under the prior predictive $m(\mathbf{x}_{n})$ and under $\theta_{\mathrm{true}}$ as

[TABLE]

Under Assumptions 3.1 we then have

[TABLE]

Remark 3.3**.**

The limit of $c_{\mathrm{true},n}$ is a special case of the one for $c_{n}$ by taking the delta distribution $\delta_{\theta_{\mathrm{true}}}$, centered at $\theta_{\mathrm{true}}$, as a prior.

Proof.

Let us first consider the limit of $c_{n}$. Note that the second condition of point 2 in Assumption 3.1 states that $X_{n}:=n\,u_{n}^{2}$ converges to $I_{\theta_{0}}^{-1}$ in $L^{2}$ with respect to the law $p(\theta,\mathbf{x}_{n})=\pi(\theta)\cdot p(\mathbf{x}_{n}|\theta)$. Since $X_{n}$ is not dependent on $\theta$, $I_{\theta_{0}}$ is not dependent on $\mathbf{x}_{n}$ and the marginals of $p(\theta,\mathbf{x}_{n})$ are $\pi(\theta)$ and $m(\mathbf{x}_{n})$ we conclude that

[TABLE]

and similar $\mathbb{E}_{\mathbf{x}_{n}\sim m(\mathbf{x}_{n})}[X_{n}]\rightarrow\mathbb{E}_{\theta\sim\pi(\theta)}[I_{\theta_{0}}^{-1}]$. From this we obtain the claimed limit of $c_{n}$:

[TABLE]

For the second part of the claim we use that by the first condition in point 2 of Assumption 3.1 we have $L^{2}$ convergence of $X_{n}$ conditional on $\theta_{\mathrm{true}}$ from which we obtain indeed

[TABLE]

where we used on the right hand side that $I_{\theta_{\mathrm{true},0}}$ does not depend on $\mathbf{x}_{n}$. ∎

We see from Lemma 3.2 that the variation of $u_{n}^{2}$ under the marginal $m(\mathbf{x}_{n})$ is behaving in a different manner as under the true parameter $\theta_{\mathrm{true}}$. The object $\Delta u_{n}^{2}=(\mathrm{Var}_{\mathbf{x}_{n}\sim m(\mathbf{x}_{n})}(u_{n}^{2}))^{1/2}$ decays at the same order as $\overline{u_{n}^{2}}=\mathbb{E}_{\mathbf{x}_{n}\sim m(\mathbf{x}_{n})}[u_{n}^{2}]$ whereas, being conditional on a parameter, the standard deviation $(\mathrm{Var}_{\mathbf{x}_{n}\sim p(\mathbf{x}_{n}|\theta_{\mathrm{true}})}(u_{n}^{2}))^{1/2}$ decays faster than $\mathbb{E}_{\mathbf{x}_{n}\sim p(\mathbf{x}_{n}|\theta_{\mathrm{true}})}[u_{n}^{2}]$. The distributions $m(\mathbf{x}_{n})$ and $p(\mathbf{x}_{n}|\theta_{\mathrm{true}})$ result in quite different asymptotic behavior for the moments of the posterior variance. We will see in Theorem 3.5 below that these different asymptotic properties will provoke some phase transition with respect to $k$ in the limit $\varepsilon\rightarrow 0$.

As a first consequence of the observations from Lemma 3.2 we will now see that the variance term $k\,\Delta u_{n}^{2}$ in the VPVC will strongly affect the sample size even as $\varepsilon\rightarrow 0$.

Lemma 3.4** (VPVC sample size for small $\varepsilon$).**

Provided Assumption 3.1 is true, we have

[TABLE]

Proof.

We have used already in the first part of the proof of Lemma 3.2 that $n\overline{u_{n}^{2}}\rightarrow\mathbb{E}_{\theta\sim\pi(\theta)}[I_{\theta_{0}}^{-1}]$ so that applying in addition the convergence $c_{n}\rightarrow\gamma$ from Lemma 3.2 we arrive at

[TABLE]

Next, observe that since $\overline{u_{n}^{2}}>0$ by point 3 of Assumption 3.1 this implies that we can find a constant $c>0$ such that for any $n$ we have $\frac{c}{n}<\overline{u_{n}^{2}}+k\Delta u_{n}^{2}$, from which we get $n_{\varepsilon}>c\cdot\varepsilon^{-2}$ and in particular

[TABLE]

Let us rewrite the claim of the lemma as

[TABLE]

From (18), and the choice of $n_{\varepsilon}$, we already have one inequality of (20):

[TABLE]

To show the opposite inequality we argue by contradiction. Suppose that there is a $\delta>0$ such that

[TABLE]

Define $m_{\varepsilon}=\lfloor s_{\infty}\varepsilon^{-2}\rfloor$, the largest integer less equal $s_{\infty}\varepsilon^{-2}$, and note that for $\varepsilon$ small enough there is an integer $n_{\varepsilon}^{\prime}$ such that the following nested inequality is true

[TABLE]

Choose further $\varepsilon$ small enough such that for any $n>m_{\varepsilon}$ we have $s_{n}/s_{\infty}<(1+\delta/2)$. But then the following inequality holds:

[TABLE]

By (22) we have thus found a smaller integer $1\leq n_{\varepsilon}^{\prime}<n_{\varepsilon}$ that satisfies the VPVC (14), which contradicts the choice of $n_{\varepsilon}$. ∎

For the setup considered in Section 2.2 we sketched the convergence of the object from Lemma 3.4 for various priors in Figure 7. The formula from Lemma 3.4 allows for a few interesting observations.

First, we see that asymptotically the prior $\pi(\theta)$ has an impact on the sample size through $\mathbb{E}_{\theta\sim\pi(\theta)}[I_{\theta_{0}}^{-1}]$ (the expectation of the rescaled B.-v.-Mises limit of the posterior variance) and through the coefficient of variation $\gamma$. The higher the latter the more influence will the choice of $k$ have on the sample size. Moreover, comparing the sample sizes $n_{\varepsilon}$ from the VPVC and $\tilde{n}_{\varepsilon}$ from the APVC ($k=0$) we see that taking the variance into account will increase the sample size by a factor of $\frac{n_{\varepsilon}}{\tilde{n}_{\varepsilon}}\simeq 1+k\cdot\gamma$. Finally, note that we do not need any explicit expression for $u_{n}^{2}$ when using the asymptotic expression of $n_{\varepsilon}$. Given the Fisher information $I_{\theta_{0}}$ of the sampling distribution and any prior $\pi(\theta)$ we can directly compute the sample size for small $\varepsilon$ without any need of computing the posterior distribution.

We will now turn to the main result of this section. In Section 2 we have evaluated the “success” of the SSD by evaluating how often we will have $u_{n_{\varepsilon}}^{2}<\varepsilon^{2}$ on the actual dataset, recall for instance Figure 4(b). In the following we want to do this for $\varepsilon\rightarrow 0$ in a purely generic setup by assuming that the data $\mathbf{x}_{n}$ follows the distribution $p(\mathbf{x}_{n}|\theta_{\mathrm{true}})$ with the true parameter $\theta_{\mathrm{true}}$. We have already observed in Lemma 3.2 that conditioning on a parameter like $\theta_{\mathrm{true}}$ leads to a different asymptotic behavior than considering the marginal $m(\mathbf{x}_{n})$. The following theorem shows the consequence of this disparity concerning the success of the SSD.

Theorem 3.5** (VPVC becomes conservative for large $k$).**

Assume that Assumptions 3.1 hold and define

[TABLE]

We then have for any $k>k_{\ast}$

[TABLE]

In particular, we have for any SSD build on the VPVC (14) with such a $k$

[TABLE]

Moreover, $k_{\ast}$ is optimal in the sense that for any $0<k<k_{\ast}$ we have

[TABLE]

Proof.

We introduce $\rho=\frac{I_{\theta_{\mathrm{true},0}}^{-1}}{\mathbb{E}_{\theta\sim\pi(\theta)}[I_{\theta_{0}}^{-1}]}-1$ so that $k_{\ast}$ can be written as $k_{\ast}=\max(\rho,0)/\gamma$ with $\gamma$ as in Lemma 3.2. Similar as in the proof of Lemma 3.2 we see that

[TABLE]

Let us first show (24). We can rewrite the inequality inside the probability as

[TABLE]

The expression on the right hand side becomes finally positive when $n$ is high enough. Indeed, we can we reshape it, using point 3 of Assumption 3.1 and the object $c_{n}$ from Lemma 3.2, as

[TABLE]

where $h_{n}$ satisfies $\lim_{n\rightarrow\infty}h_{n}=-\rho+k\cdot\gamma>0$ due to Lemma 3.2, (27) and the choice $k>k_{\ast}=\max(\rho,0)/\gamma$. For $n$ large enough such that the right hand side of (28) is positive, we can apply Chebyshev’s inequality, which yields

[TABLE]

where we used once more (27) and that we know from Lemma 3.2 that $c_{\mathrm{true},n}$ converges to 0. From (24) we immediately obtain (25) by the definition of $n_{\varepsilon}$ and the fact that $n_{\varepsilon}\rightarrow\infty$, due to Lemma 3.4. To show (26) we reshape the expression inside the parentheses of (26) into

[TABLE]

The rest of the argument then follows along the lines of the first part of the proof. ∎

Theorem 3.5 reveals some sort of phase transition. For a $k$ smaller than some threshold $k_{\ast}$ the squared uncertainty $u_{n}^{2}$ will asymptotically be above $u_{n}^{2}+k\Delta u_{n}^{2}$, while for $k>k_{\ast}$ we will have $u_{n}^{2}<\overline{u_{n}^{2}}+k\Delta u_{n}^{2}$ with probability almost 1 for large $n$ (or small $\varepsilon$). This effect is shown in Figure 8 for the Football goal data from section 2.1 and a fixed prior.

The behavior predicted by Theorem 3.5 might be surprising at first glance: recall that $\overline{u_{n}^{2}}$ is defined as an average and that the object $\Delta u_{n}^{2}$ is the according standard deviation. Naively one might expect that the range of $k$ times the standard deviation around the mean might always cover some percentage strictly between [math] and $1$. The reason behind the observed phenomenon is that we use one distribution, $m(\mathbf{x}_{n})$, to perform the SSD and another one, $p(\mathbf{x}_{n}|\theta_{\mathrm{true}})$, to evaluate its success. The different asymptotics between these two distributions then gives rise to the erratic behavior around $k=k_{\ast}$. The effect for a fixed $k$ and various priors is depicted by Figure A.1 in the appendix: As $\varepsilon$ gets small we get two sharply separated regions for the hyperparameters where one region has a quota of $u_{n_{\varepsilon}}^{2}<\varepsilon^{2}$ around $0\%$ while the other one approaches $100\%$. We expect that this phenomenon is not solely adherent to the VPVC. For instance, it seems reasonable that for many cases a credible interval for $u^{2}_{n}$ under $m(\mathbf{x}_{n})$ is, at least loosely, related to a certain $k$ so that similar effects are likely to arise for a framework based on other approaches using $m(\mathbf{x}_{n})$. Investigating such a point further might be an interesting subject for future research.

We have seen in Theorem 3.5 that for small $\varepsilon$ the success of the SSD crucially depends on having $k>k_{\ast}$. Now, $k_{\ast}$ is dependent on the true parameter $\theta_{\mathrm{true}}$ and thus unknown in practice. For the prior used for Figure 8 we have a $k_{\ast}$ around $0.21$ and therefore much smaller than the $k=2$ used in the analysis of Section 2.1. In general one might choose for an area $\Theta$ in the domain of $\pi(\theta)$ as an upper bound for $k_{\ast}$

[TABLE]

Taking a single standard deviation around the prior mean for $\pi(\theta)$ and $\pi(\sigma^{2})$ respectively we get for the prior from Figure 8 $k_{\ast}^{\tiny\mbox{up. b.}}=1.2$ and for the one marked by the black cross in Figure 6(b) $k_{\ast}^{\tiny\mbox{up. b.}}=1.0$. For two standard deviations we obtain 2.2 and 2.0 respectively. All these values are well above the actual values of $k_{\ast}$ which are 0.2 and 0.7 for these two examples and below or at least close to the value of $k=2$ we applied throughout Section 2.

4 Conclusions

We discussed a Bayesian criterion for sample size determination, based on the prior predictive, which we called the variation of posterior variance criterion (VPVC). Compared with the classical average posterior variance criterion (APVC) this criterion leads to a better compliance with the objective to restrain the uncertainty by some $\varepsilon$, while still providing explicit expressions in contrast to a full treatment of the law of the uncertainty under the prior predictive. In particular this allows to treat the asymptotic behavior for small $\varepsilon$ in a generic manner and thus to enhance the understanding of sample size methods based on the prior predictive.

Using two different datasets we discussed the dependency of the sample size determination and its success on the chosen prior and the true parameter and deduced concepts on how to choose the prior to lead the sample size determination more likely to a success. In particular, we observed that a part of these strategies cannot be applied for the APVC but only for methods such as the VPVC, that take higher moments with respect to the prior predictive into account.

Finally, we gave some results concerning the behavior of the VPVC for $\varepsilon\rightarrow 0$. We proved an explicit formula for the predicted sample size in this regime and showed that there is an exact limit for the portion of the variance w.r.t. prior predictive that can be used in order to guarantee the success of the sample size determination in this limit.

Appendix A Effect of decreasing $\varepsilon$ on $u^{2}_{n_{\varepsilon}}<\varepsilon^{2}$