On one-sample Bayesian tests for the mean

TL;DR

This paper introduces a Bayesian method for one-sample z- and t-tests based on Kullback-Leibler divergence, providing evidence for the null hypothesis and avoiding classical paradoxes, with theoretical and practical validation.

Contribution

It presents a novel Bayesian testing approach using KL divergence and relative belief ratios, offering evidence for the null and addressing limitations of existing methods.

Findings

Method provides evidence in favor of the null hypothesis.

Avoids classical paradoxes like Lindley's paradox.

Demonstrated effectiveness through examples.

Abstract

This paper deals with a new Bayesian approach to the standard one-sample $z$- and $t$- tests. More specifically, let $x_1,\ldots,x_n$ be an independent random sample from a normal distribution with mean $\mu$ and variance $\sigma^2$. The goal is to test the null hypothesis $\mathcal{H}_0: \mu=\mu_1$ against all possible alternatives. The approach is based on using the well-known formula of the Kullbak-Leibler divergence between two normal distributions (sampling and hypothesized distributions selected in an appropriate way). The change of the distance from a priori to a posteriori is compared through the relative belief ratio (a measure of evidence). Eliciting the prior, checking for prior-data conflict and bias are also considered. Many theoretical properties of the procedure have been developed. Besides it's simplicity, and unlike the classical approach, the new approach possesses attractive and distinctive features such as giving evidence in favor of the null hypothesis. It also avoids several undesirable paradoxes, such as Lindley's paradox that may be encountered by some existing Bayesian methods. The use of the approach has been illustrated through several examples.

Tables

Table 1. Table 1: The tests results about the dental anxiety example.

Test	Values	Decision
Distance: RB (Strength)	4.7800(1)	Accept the null hypothesis
Direct: RB (Strength)	4.8466(0.4341)	Accept the null hypothesis
Rouder et. al. (2009): BF	0.2747	Accept the null hypothesis
$t$-test: p-value	0.7517	Fail to reject the null hypothesis

Table 2. Table 2: The tests results about the age at which children start walking.

Test	Values	Decision
Distance: RB (Strength)	6.7480(1)	Accept the null hypothesis
Direct: RB (Strength)	6.5420(0.4479)	Accept the null hypothesis
Rouder et. al. (2009): BF	1.467772	Reject the null hypothesis
$t$-test: p-value	.0489	Reject the null hypothesis

Table 3. Table 3: The tests results about the sugar production.

Test	Values	Decision
Distance: RB (Strength)	0.4680(0.0442)	Reject the null hypothesis
Direct: RB (Strength)	0.4355(0.0189)	Reject the null hypothesis
Rouder et. al. (2009): BF	129.1731	Reject the null hypothesis
$t$-test: p-value	0.0002	Reject the null hypothesis

Equations

$$z=\frac{\bar{x}-{\mu }_1}{\sigma /\sqrt{n}},$$

$$t=\frac{\bar{x}-{\mu }_1}{s/\sqrt{n}},$$

$$\pi (\theta |x)=\frac{f_{\theta }(x)\pi (\theta )}{m(x)},$$

$$m(x)=\int f_{\theta }(x)\pi (\theta )d\theta$$

$$R{B}_{\Psi }({\psi }_0|x)={\pi }_{\Psi }({\psi }_0|x)/{\pi }_{\Psi }({\psi }_0),$$

$$R{B}_{\Psi }({\psi }_0|x)=m_T(T(x)|{\psi }_0)/m_T(T(x)),$$

$${\Pi }_{\Psi }(R{B}_{\Psi }(\psi |x)\le R{B}_{\Psi }({\psi }_0|x)|x).$$

$$M(R{B}_{\Psi }({\psi }_0|x)\le 1|{\psi }_0)$$

$$M(R{B}_{\Psi }({\psi }_0|x)\ge 1|{\psi }_0^{\prime })$$

$$M_T\left(m_T(t)\le m_T(T(x))\right),$$

$${\Pi }_T\left({\pi }_0(\theta )\le {\pi }_0({\theta }_{\text{true}})\right),$$

$${\mu }_x=\frac{n{\lambda }_0^2}{n{\lambda }_0^2+1}\bar{x}+\frac{1}{n{\lambda }_0^2+1}{\mu }_0\text{ and }{\sigma }_x^2=\frac{{\lambda }_0^2}{n{\lambda }_0^2+1}.$$

$$RB(\mu |x)=\frac{\pi (\mu |T(x))}{\pi (\mu )}=\frac{\pi (\mu )f(T(x))/m_T(T(x))}{\pi (\mu )}=\frac{f(T(x))}{m_T(T(x))}.$$

$$m_T(T(x))=\sqrt{\frac{n}{2\pi {\sigma }^2(1+n{\lambda }_0^2)}}\exp \left(-\frac{n}{2}\frac{(\bar{x}-{\mu }_0{)}^2}{{\sigma }^2(n{\lambda }_0^2+1)}\right).$$

$$RB(\mu |x)=\sqrt{1+n{\lambda }_0^2}\exp \left(-\frac{n}{2{\sigma }^2}\left[(\bar{x}-\mu {)}^2-\frac{(\bar{x}-{\mu }_0{)}^2}{(n{\lambda }_0^2+1)}\right]\right).$$

$$\displaystyle\Pi\Bigg{(}\exp\left(-\frac{n}{2\sigma^{2}}\left[(\bar{x}-\mu)^{2}-\frac{(\bar{x}-\mu_{0})^{2}}{(n\lambda_{0}^{2}+1)}\right]\right)$$

$$\Pi \left(RB(\mu |x)\le RB({\mu }_1|x)|x\right)$$

$$d(P,Q)=\int p(x)\log \left(\frac{p(x)}{q(x)}\right)dx.$$

$$d(P,Q)=\log (\frac{{\sigma }_1}{{\sigma }_2})+\frac{1}{2{\sigma }_2^2}\left[{\sigma }_1^2+({\mu }_1-{\mu }_2{)}^2\right]-\frac{1}{2}.$$

$$d(P^{\text{prior}},Q)=\frac{(\mu -{\mu }_1{)}^2}{2{\sigma }^2}.$$

$$d(P^{\text{post}},Q)=\frac{({\mu }_x-{\mu }_1{)}^2}{2{\sigma }^2}.$$

$$d(P^{\text{Post}},Q)\stackrel{a.s.}{\to }c>0.$$

$$M_T\left(m_T(t)\le m_T(\bar{x})\right)=2\left(1-\Phi \left(\left|\bar{x}-{\mu }_0\right|/{\left({\lambda }_0^2{\sigma }^2+{\sigma }^2/n\right)}^{0.5}\right)\right),$$

$${\widehat{RB}}_D(d|x)=M\{{\hat{F}}_D({\hat{d}}_{(i+1)/M}|x)-{\hat{F}}_D({\hat{d}}_{i/M}|x)\},$$

$$\sum _{\{i\ge i_0:{\widehat{RB}}_D({\hat{d}}_{i/M}|x)\le {\widehat{RB}}_D(0|x)\}}({\hat{F}}_D({\hat{d}}_{(i+1)/M}|x)-{\hat{F}}_D({\hat{d}}_{i/M}|x)).$$

$$R{B}_D([0,d_{i_0/M})|x)\stackrel{a.s.}{\to }1/D{P}_D([0,d_{i_0/M})),$$

$$R{B}_D([d_{i/M},d_{(i+1)/M})|x)\stackrel{a.s.}{\to }0\text{ whenever }i\ge i_0,$$

$$D{P}_D(R{B}_D(d|x)\le R{B}_D(0|x)|x)\stackrel{a.s.}{\to }1,$$

$$\frac{1}{{\sigma }^2}\sim {\text{gamma}}_{rate}({\alpha }_0,{\beta }_0)$$

$$\mu |{\sigma }^2\sim \mathcal{N}({\mu }_0,{\lambda }_0^2{\sigma }^2),$$

$$\displaystyle\frac{1}{\sigma^{2}}\big{|}x_{1},...,x_{n}\sim\text{gamma}_{rate}(\alpha_{0}+\frac{n}{2},\beta_{x}),$$

$$\mu |{\sigma }^2,x_1,...,x_n\sim \mathcal{N}\left({\mu }_x,(n+\frac{1}{{\lambda }_0^2}{)}^2{\sigma }^2\right)$$

Full text

Keywords: Hypothesis testing, Kullbak-Leibler divergence, one-sample $t$-test, one-sample $z$-test, relative belief inferences.

MSC 2000 62F03, 62F15

1 Introduction

The one-sample hypothesis testing is a primary topic in any introductory statistics course. It involves the selection of a reference value $\mu_{1}$ for the (unknown) population mean $\mu$. More specifically, let $x=(x_{1},\ldots,x_{n})$ be an independent random sample taken from $N(\mu,\sigma^{2})$, where $\sigma^{2}$ is the population variance. The interest is to test the hypothesis $H_{0}:\mu=\mu_{1}$, where $\mu_{1}$ is a given real number. Within the classical frequentist frame work, if $\sigma$ is known, then the $z$-test is commonly used for testing $H_{0}$ against the two-sided alternative $H_{1}:\mu\neq\mu_{1}$. The test statistics in this case is

[TABLE]

where $\bar{x}$ is the sample mean. For a significant level $\alpha$, the critical value $Z_{\alpha/2}$ is defined to be the $1-\alpha/2$ quantile of the standard normal distribution. Also, the $p$-value is equal to $2P(Z>|z|)$, where $Z$ has the standard normal distribution. Then, $H_{0}$ is rejected if $|z|\geq Z_{\alpha/2}$ or the $p$-value less than $\alpha$. On the other hand, if $\sigma$ is unknown, then the test statistic is

[TABLE]

where $s$ is the sample standard deviation. For a test with significant level $\alpha$, let $t_{n-1,\alpha/2}$ be the $1-\alpha/2$ quantile of the $t$ distribution with $n-1$ degrees of freedom. The two sided $p$-value is equal to $2P(T>|t|)$, where $T$ has the t-distribution with $n-1$ degrees of freedom. Similar to the $z$-test, $H_{0}$ is rejected if $|t|\geq t_{n-1,\alpha/2}$ or the $p$-value is less than $\alpha$.

While the above approach for hypothesis testing is well-known and stable, it is difficult to find an alternative Bayesian counterpart in the literature. An exception includes the work of Rouder, Speckman, Sun, and Morey (2009) who proposed a Bayesian test, where $\sigma$ is unknown, using the Bayes factor (ratio of the marginal densities of the two models; Kass and Raftery, 1995). They placed the Jeffreys prior for $\sigma$ and the Cauchy prior on $\mu/\sigma$. They provided a web-based program (c.f. pcl.missouri.edu) in order to facilitate the use of their test. Remarkably, the authors mentioned detailed criticisms of using the $p$-values in hypothesis testing. For example, they indicated that the $p$-values do not allow researchers to state evidence for the null hypothesis. They also overstate the evidence against the null hypothesis. Although the $p$-value converges to zero as the sample size increases when the null hypothesis is false which is a desirable feature, the $p$-values are all equally likely and uniformly distributed between 0 and 1 when null is true. This distribution holds regardless of the sample size which means that increasing the sample size in this case will not help gaining evidence for the null hypothesis. In fact, this reflects Fisher’s sight that the null hypothesis can only be rejected and never accepted. Other relevant work, but in the two-sample problem set up, includes Gönen, Johnson, Lu and Westfall (2005) and Wang and Lui (2016). For more recent articles about the limitations of using $p$-values in hypotheses testing, we refer the reader to Evans (2015), Wasserstein and Lazar (2016), and references therein.

Unlike the previous work, the hyperparameters of the prior in the new approached Bayesian are elicited and tested against prior-data conflict and against being biased. For this, two elicitation algorithms developed by Evans (2015, 2018) are considered. In fact, the success of any Bayesian approach depends significantly on a proper selection of the hyperparameters of the prior. Part of the elicitation process involves checking the elicited prior for the prior-data conflict and the bias (see Section 2). Then the concentration of the distribution of the Kullbak-Leibler divergence between the prior and the model of interest is compared to that between the posterior and the model. If the posterior is more concentrated about the hypothesized distribution than the prior, then this is evidence in favor of the null hypothesis and if the posterior is less concentrated then this is evidence against the null hypothesis. This comparison is made via a relative belief ratio, which measures the evidence in the observed data for or against the null. A measure of the strength of this evidence is also provided. So, the methodology is based on a direct measure of statistical evidence. We point out that, relative belief ratios have been recently used in problems that involve goodness of fit test and model checking. See, for example, Al-Labadi (2018), Al-Labadi and Evans (2018) and Al-Labadi, Zeynep and Evans (2017, 2018) and Evans and Tomal (2018).

The proposed method brings many advantages to the problem of hypothesis testing. Besides its simplicity, and unlike the classical approach, the new approach possesses attractive and desirable features such as giving evidence in favor of the null hypothesis. Also, checking the prior for bias and prior-data conflict permits avoid several undesirable paradoxes, such as Lindley’s paradox that may be encountered by the standard Bayesian methods that are based, for instance, on the Bayes factor (Evans, 2015).

The remainder of this paper is organized as follows. A general discussion about the relative belief ratio is given in Section 2. The definition and some fundamental properties of the Dirichlet process are presented in Section 3. In Section 4, an explicit expression to compute Anderson-Darling distance between the Dirichlet process and its base measure is derived. In Section 5, a Bayesian nonparametric test for assessing multivariate normality is discussed and some of its relevant properties are developed. A computational algorithm to calculate the relative belief ratio for the implementation of the proposed test is developed in Section 6. In Section 7, the performance of the proposed test is established via four simulated examples and two real data sets. Finally, some concluding remarks are given in Section 8. All technical proofs are included in the supplementary material.

2 Inferences Using Relative Belief

Suppose we have a statistical model that is given by the density function $f_{\theta}(x)$ (with respect to some measure), where $\theta$ is an unknown parameter that belongs to the parameter space $\Theta$. Let $\pi(\theta)$ be the prior distribution of $\theta$. After observing the data $x$, by Bayes’ theorem, the posterior distribution of $\theta$ is given by the density

[TABLE]

where

[TABLE]

is the prior predictive density of the data.

Suppose that the interest is to make inference about an arbitrary parameter $\psi=\Psi(\theta)$. Let $\Pi_{\Psi}$ denote the prior measure of $\psi$ with density $\pi_{\Psi}$. Let the corresponding posterior measure and density of $\psi$ be $\Pi_{\psi}(\cdot\,|\,x)$ and $\pi_{\Psi}(\cdot\,|\,x),$ respectively. The relative belief ratio for a hypothesized value $\psi_{0}$ of $\psi$ is defined by $RB_{\Psi}(\psi_{0}\,|\,x)=\lim_{\delta\rightarrow 0}\Pi_{\Psi}(N_{\delta}(\psi_{0}\,)|\,x)/\Pi_{\Psi}(N_{\delta}(\psi_{0}\,))$, where $N_{\delta}(\psi_{0}\,)$ is a sequence of neighbourhoods of $\psi$ converging nicely (see, for example, Rudin (1974)) to $\psi$ as $\delta\rightarrow 0.$ When $\pi_{\Psi}$ and $\pi_{\Psi}(\cdot\,|\,x)$ are continuous at $\psi,$

[TABLE]

is the ratio of the posterior density to the prior density at $\psi_{0}.$ That is, $RB_{\Psi}(\psi_{0}\,|\,x)$ is measuring how beliefs have changed that $\psi_{0}$ is the true value from a priori to a posteriori. Baskurt and Evans (2013) proved that

[TABLE]

where $T$ is a minimal sufficient statistic of the model and $m_{T}$ is the prior predictive density of $T$. The previous authors referred to (1) as the Savage-Dickey ratio. It is to be noted that a relative belief ratio is similar to a Bayes factor (Kass and Raftery, 1995), as both are measures of evidence, but the latter measures it via the change in an odds ratio. A discussion about the relationship between relative belief ratios and Bayes factors is detailed in (Baskurt and Evans, 2013). More specifically, when a Bayes factor is defined via a limit in the continuous case, the limiting value is the corresponding relative belief ratio.

By a basic principle of evidence, $RB_{\Psi}(\psi_{0}\,|\,x)>1$ means that the data led to an increase in the probability that $\psi_{0}$ is correct, and so there is evidence in favour of $\psi_{0},$ while $RB_{\Psi}(\psi_{0}\,|\,x)<1$ means that the data led to a decrease in the probability that $\psi_{0}$ is correct, and so there is evidence against $\psi_{0}$. Clearly, when $RB_{\Psi}(\psi_{0}\,|\,x)=1$, then there is no evidence either way.

It is also important to calibrate whether this is strong or weak evidence for or against $\mathcal{H}_{0}$. As suggested in Evans (2015), a useful calibration of $RB_{\Psi}(\psi_{0}\,|\,x)$ is obtained by computing the tail probability

[TABLE]

One way to view (2) is as the posterior probability that the true value of $\psi$ has a relative belief ratio no greater than that of the hypothesized value $\psi_{0}.$ When $RB_{\Psi}(\psi_{0}\,|\,x)<1,$ there is evidence against $\psi_{0},$ then a small value for (2) indicates a large posterior probability that the true value has a relative belief ratio greater than $RB_{\Psi}(\psi_{0}\,|\,x)$ and so there is strong evidence against $\psi_{0}.$ When $RB_{\Psi}(\psi_{0}\,|\,x)>1,$ there is evidence in favour of $\psi_{0},$ then a large value for (2) indicates a small posterior probability that the true value has a relative belief ratio greater than $RB_{\Psi}(\psi_{0}\,|\,x))$. Therefore, there is strong evidence in favour of $\psi_{0},$ while a small value of (2) only indicates weak evidence in favour of $\psi_{0}.$

One of the key concerns with Bayesian inference methods is that the prior can bias the analysis. Following Evans (2015), let $M(\cdot|\psi)$ denote the conditional prior predictive distribution of the data given that $\Psi(\theta)=\psi$, so $M(A|\psi)=\int_{\Theta}\left(\int_{A}f_{\theta}(x)dx\right)\pi(\theta|\psi)d\theta$ is the conditional prior probability that the data is in the set $A$. The bias against $H_{0}:\Psi(\theta)=\psi_{0}$ can be measured by computing

[TABLE]

and this is the prior probability that evidence will be obtained against $H_{0}$ when it is true. If the bias against $H_{0}$ is large, subsequently reporting, after seeing the data, then there is evidence against $H_{0}$ is not convincing.On the other hand, the bias in favor of $H_{0}$ is given by

[TABLE]

for values $\psi_{0}\neq\psi^{\prime}_{0}$ such that the difference between $\psi_{0}$ and $\psi^{\prime}_{0}$ represents the smallest difference of practical importance; note that this tends to decrease as $\psi^{\prime}_{0}$ moves farther away from $\psi_{0}$. When the bias in favor is large, subsequently reporting, after seeing the data, then the is evidence in favor of $H_{0}$ is not convincing.

Another concern regarding priors is to measure the compatibility between the prior and the data. A chosen prior may be incorrect by being strongly contradicted by the data (Evans, 2015). A possible contradiction between the data and the prior is referred to as a prior-data conflict. If the prior primarily places its mass in a region of the parameter space where the data suggest the true value does not lie, then there is a prior-data conflict (Evans and Moshonov, 2006). That is, prior-data conflict will occur whenever there is only a tiny overlap between the effective support regions of the model and the prior. In such situation, we must be concerned about what the effect of the prior is on the analysis (Evans, 2015). Methods for checking the prior in previous sense are developed in Evans and Moshonov (2006). See also Nott, Xueou, Evans, and Engler (2016) and Nott, Seah, AL-Labadi, Evans, Ng and Englert (2019). The basic method for checking the prior involves computing the probability

[TABLE]

where $T$ is a minimal sufficient statistic of the model and $M_{T}$ is the prior predictive probability measure of $T$ with density $m_{T}$. The value of (5) simply serves to locate the observed value $T(x)$ in its prior distribution. If (5) is small, then $T(x)$ lies in a region of low prior probability, such as a tail or anti-mode, which indicates a conflict. The consistency of this check follows from Evans and Jang (2011) where it is proven that, under quite general conditions, (5) converges to

[TABLE]

as the amount of data increases, where $\theta_{\text{true}}$ is the true value of the parameter. If (6) is small, then $\theta_{\text{true}}$ lies in a region of low prior probability which implies that the prior is not appropriate.

3 A Bayesian Alternative to the One-Sample $z-$Test

3.1 The Approach

Let $x=(x_{1},\ldots,x_{n})$ be an independent random sample from $N(\mu,\sigma^{2})$, where $\sigma^{2}$ is known. The goal is to test the hypothesis $H_{0}:\mu=\mu_{1}$, where $\mu_{1}$ is a given real number. The approach here is Bayesian. First we construct a prior $\pi(\mu)$ on $\mu$. Let $\pi(\mu)$ be $N(\mu_{0},\lambda^{2}_{0}\sigma^{2})$, where $\mu_{0}$ and $\lambda_{0}^{2}$ are known hyperparameters and selected through the elicitation algorithms covered in Section 3.2. Thus, the posterior distribution of $\mu$ given $x_{1},\ldots,x_{n}$ is $\pi(\mu|x_{1},\ldots,x_{n})=N(\mu_{x},\sigma_{x}^{2})$, where

[TABLE]

To proceed for the test using the relative belief ratio, there are two possible approaches. The first one is based on a direct computation of the relative belief ratio $RB(\mu_{1}|x)$ and its strength. This approach has been initiated in Baskurt and Evans (2013) with $\sigma^{2}=1$ and $\mu_{1}=0$ when discussing the Jeffrey-Lindely paradox. To find $RB(\mu|x)$, notice that

[TABLE]

The minimal sufficient statistics for $\mu$ is $T(x)=\bar{x}\sim N(\,\mu,\sigma^{2}/n)$. Since $T(x)=\bar{x}=(\bar{x}-\mu)+\mu$, where $\bar{x}-\mu\sim N(0,\sigma^{2}/n)$ independent of $\mu\sim N(\mu_{0},\lambda^{2}_{0}\sigma^{2})$, it follows the prior predictive distribution of $T(x)$ is $N(\mu_{0},\lambda^{2}_{0}\sigma^{2}_{0}+\sigma^{2}/n)$. That is,

[TABLE]

Thus,

[TABLE]

For the strength, we have $\Pi\left(RB(\mu|x)\leq RB(\mu_{1}|x)|x\right)=$

[TABLE]

where $\mu_{x}$ and $\sigma_{x}$ are defined in (7). After minor simplification we have,

[TABLE]

Similar to the conclusion in Baskurt and Evans (2013), as $\lambda_{0}^{2}\to\infty$ in (9), $\Pi\left(RB(\mu|x)\leq RB(\mu_{1}|x)|x\right)\to 2\left(1-\Phi(\sqrt{n}|\bar{x}-\mu_{1}|/\sigma)\right)$, which converges in distribution to $2\left(1-\Phi(|z|)\right)$ when $\mu=\mu_{1}$, by the central limit theorem and the continuous mapping theorem, where $z$ is the standard normal random variable. Hence, when $\mu=\mu_{1}$ (i.e. $H_{0}$ is not rejected), the strength has an asymptotically uniform distribution on $(0,1)$. On the other hand, we have $\Pi\left(RB(\mu|x)\leq RB(\mu_{1}|x)|x\right)$ converges to 0 almost surely (a.s.) when $\mu\neq\mu_{1}$, since $\bar{x}\to\mu$ almost surely.

As for the second approach, we compute the KL distance between the hypothesized distribution and the prior/posterior distributions. The change of the distance from a priori to a posteriori is compared through the relative belief ratio. Then, we give a brief summary about the KL distance. In general, the KL distance (sometimes called the entropy distance) between two continuous cumulative distribution functions (cdf’s) $P$ and $Q$ with corresponding probability density functions (pdf’s) $p$ and $q$ (with respect to Lebesgue measure) is defined by

[TABLE]

It is well-known that $d_{KL}(P,Q)\geq 0$ and the equality holds if and only if $p=q$. However, it is not symmetric and does not satisfy the triangle inequality (Cover and Thomas, 1991). In particular, the KL divergence between the two normal distributions $P=N(\mu_{1},\sigma_{1}^{2})$ and $Q=N(\mu_{2},\sigma_{2}^{2})$ is given by (Duchi, 2014)

[TABLE]

Set $P^{\text{prior}}=N(\mu,\sigma^{2})$ and $Q=N(\mu_{1},\sigma^{2})$. It follows that from (10) that

[TABLE]

Also the KL divergence between $P^{\text{post}}=N(\mu_{x},\sigma^{2})$ and Q is

[TABLE]

Note that, as $n\rightarrow\infty$, by the strong law of large numbers, $\mu_{x}\overset{a.s.}{\to}\mu_{\text{true}}$, where $\mu_{\text{true}}$ is the true value of $\mu$. Thus, by (12), if $H_{0}$ is true, we have $d(P^{\text{post}},Q)\overset{a.s.}{\to}0$. On the other hand, if $H_{0}$ is not true, then

[TABLE]

What follows is that, if $H_{0}$ is true, then that distribution of $d(P^{\text{Post}},Q)$ should be more concentrated about [math] than $d(P^{\text{Prior}},Q)$. So, the proposed test includes a comparison of the concentrations of the prior and posterior distributions of the KL divergence via a relative belief ratio based on the interpretation as discussed in  Section 2.

3.2 Elicitation of the Prior

The success of methodology is influenced significantly by the choice of the hyperparameters $\mu_{0}$ and $\lambda_{0}$. Inappropriate values of the hyperparameters can lead to a failure in computing $d$. To elicit proper values of the hyperparameters, we consider the method developed in Evans and Tomal (2018). Suppose that it is known with virtual certainty, based on the knowledge of the basic measurement being taken, that $\mu$ will lie in the interval $(a,b)$ for specified values $a\leq b$. Here, virtual certainty is interpreted as $P(a\leq\mu\leq b)\geq\gamma$, where $\gamma$ is a large probability like 0.999. If $\mu_{0}=(a+b)/2$, then after some simple algebra, $\lambda_{0}=(b-a)/(2\sigma\Phi^{-1}((1+0.999)/2))$.

3.3 Checking for Prior-Data Conflict

As pointed in Section 3.1, the minimal sufficient statistics for $\mu$ is $T(x)=\bar{x}$ with the prior predictive distribution of $T(x)$ is $N(\mu_{0},\lambda^{2}_{0}\sigma^{2}_{0}+\sigma^{2}/n)$. Thus,

[TABLE]

where $M_{T}$ is defined as in (5). Recall that, if (14) is small, then this indicates a prior-data conflict and no prior-data conflict otherwise. It is true that prior-data conflict can be avoided by increasing $\lambda_{0}$ (i.e. making the prior diffuse), however, as pointed in Evans (2018), this is not an appropriate approach as it will induce bias into the analysis. Thus, by (14), when $\bar{x}_{0}$ lies in the tail of its prior distribution, we have a prior-data conflict. Note that, as $n\to\infty,$ $(\ref{conflict1})\overset{a.s.}{\to}2\left(1-\Phi\left(\left|\mu_{\text{true}}-\mu_{0}\right|/\left(\lambda_{0}\sigma\right)\right)\right)$.

3.4 Checking for Bias

The bias against the hypothesis $H_{0}:\mu=\mu_{1}$ is measured by computing (3) with $\psi_{0}=\mu_{1}$ and $RB(\mu_{1}|x)$ as in (8). Note that, since the prior is centered at $\mu_{1}$, there is never a strong bias against $H_{0}$. On the other hand, the bias in favor of the hypothesis $H_{0}:\mu=\mu_{1}$ is measured by computing (4) with $\psi_{0}=\mu_{1}$ and $RB(\mu_{1}|x)$ as defined in (8). The interpretation of the bias was covered in Section 2.

3.5 The Algorithm

The approach will involve a comparison between the concentrations of the prior and posterior distribution of the KL divergence via a relative belief ratio, with the interpretation as discussed in Section 2. Since explicit forms of the densities of the distance are not available, the relative belief ratios need to be estimated via simulation. The following summarizes a computational algorithm for testing $H_{0}$.

Algorithm A (New $z-$Test)

1. (i)

Elicit the hyperparameters $\mu_{0}$ and $\lambda_{0}$ as described in Section 3.2. 2. (ii)

Generate $\mu$ from $N(\mu_{0},\lambda^{2}_{0}\sigma_{0}^{2})$. 3. (iii)

Compute the KL distance between $N(\mu,\sigma^{2})$ and $Q=N(\mu_{1},\sigma^{2})$ as described in (11). Denote this distance by $D$. 4. (iv)

Repeat steps (ii) and (iii) to obtain a sample of $r_{1}$ values of $D$. 5. (v)

Generate $\mu$ from $N(\mu_{x},\sigma^{2}_{x})$, where $\mu_{x}$ and $\sigma^{2}_{x}$ are defined in (7). 6. (vi)

Compute the KL distance between $N(\mu,\sigma^{2})$ and $Q=N(\mu_{1},\sigma^{2})$ as described in (12). Denote this distance by $D_{x}$. 7. (vii)

Repeat steps (v) and (vi) to obtain a sample of $r_{2}$ values of $D_{x}$. 8. (viii)

Compute the relative belief ratio and the strength as follows:

1. (a)

Closed forms of $D$ and $D_{x}$ are not available. Thus, the relative brief ration and the strength need to be estimated via approximation. Let $M$ be a positive number. Let $\hat{F}_{D}$ denote the empirical cdf of $D$ based on the prior sample in (3) and for $i=0,\ldots,M,$ let $\hat{d}_{i/M}$ be the estimate of $d_{i/M},$ the $(i/M)$-the prior quantile of $D.$ Here $\hat{d}_{0}=0$, and $\hat{d}_{1}$ is the largest value of $d$. Let $\hat{F}_{D}(\cdot\,|\,x)$ denote the empirical cdf of $D$ based on the posterior sample in (vi). For $d\in[\hat{d}_{i/M},\hat{d}_{(i+1)/M})$, estimate $RB_{D}(d\,|\,x)={\pi_{D}(d|x)}/{\pi_{D}(d)}$ by

[TABLE]

the ratio of the estimates of the posterior and prior contents of $[\hat{d}_{i/M},\hat{d}_{(i+1)/M}).$ Thus, we estimate $RB_{D}(0\,|\,x)={\pi_{D}(0|x)}/{\pi_{D}(0)}$ by $\widehat{RB}_{D}(0\,|\,x)=$ $M\widehat{F}_{D}(\hat{d}_{p_{0}}\,|\,x)$ where $p_{0}=i_{0}/M$ and $i_{0}$ are chosen so that $i_{0}/M$ is not too small (typically $i_{0}/M\approx 0.05)$. 2. (b)

Estimate the strength $DP_{D}(RB_{D}(d\,|\,x)\leq RB_{D}(0\,|\,x)\,|\,x)$ by the finite sum

[TABLE]

For fixed $M,$ as $r_{1}\rightarrow\infty,r_{2}\rightarrow\infty,$ then $\hat{d}_{i/M}$ converges almost surely to $d_{i/M}$ and (15) and (16) converge almost surely to $RB_{D}(d\,|\,x)$ and $DP_{D}(RB_{D}(d\,|\,x)\leq RB_{D}(0\,|\,x)\,|\,x)$, respectively.

The following proposition establishes the consistency of the approach as the sample size increases. So, the procedure performs correctly as the sample size increases when $\mathcal{H}_{0}$ is true. The proof follows immediately from Evans (2015), Section 4.7.1. See also AL-Labadi and Evans (2018) for a similar result.

Proposition 1

*Consider the discretization $\{[0,d_{i_{0}/M}),[d_{i_{0}/M},d_{(i_{0}+1)/M}),\ldots,$

$[d_{(M-1)/M},\infty)\}$. As $n\rightarrow\infty,$ (i) if $\mathcal{H}_{0}$ is true, then*

[TABLE]

and (ii) if $\mathcal{H}_{0}$ is false and $d_{CvM}(P,Q)\geq d_{i_{0}/M}$, then $RB_{D}([0,d_{i_{0}/M})\,|\,x)\overset{a.s.}{\rightarrow}0$ and $DP_{D}(RB_{D}(d\,|\,x)\leq RB_{D}(0\,|\,x)\,|\,x)\overset{a.s.}{\rightarrow}0.$

4 A Bayesian Alternative to the One-Sample t-Test

4.1 The Approach

In this section, we assume that $x_{1},...,x_{n}$ is an independent random sample from $\mathcal{N}(\mu,\sigma^{2})$, where $\sigma^{2}$ is unknown. The goal is to test $H_{0}:\mu=\mu_{1}$, where $\mu_{1}$ is a given real number. The first step in the approach is to construct priors on $\mu$ and $\sigma^{2}$. We will consider the following hierarchical but conjugate prior (Evans 2015, p.171):

[TABLE]

where $\mu_{0}$, $\lambda_{0}$ and $(\alpha_{0},\beta_{0})$ are hyperparameters to be specified via elicitation as it will be described in Section 4.2. The posterior distribution of $(\mu,\sigma^{2})$ is given by:

[TABLE]

where

[TABLE]

with $S^{2}=\frac{1}{n-1}\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}.$ To find $RB(\mu|x)$, notice that the minimal sufficient statistic for $T(x)=(\mu,\sigma^{2})$ is $(\bar{x},s^{2})$ with $\bar{x}\sim\mathcal{N}\left(\mu,\sigma^{2}/n\right)$ independent of $s^{2}\sim\sigma^{2}(n-1)^{-1}\chi^{2}_{n-1}$. The joint prior predictive of $T(x)=(\bar{x},s^{2})$ is given by (Evan, 2015):

[TABLE]

where $\beta_{x}$ is defined in (21). On the other hand, it can be shown that

[TABLE]

Thus,

[TABLE]

For the strength we have, $\Pi\left(RB(\mu|x)\leq RB(\mu_{1}|x)|x\right)=$

[TABLE]

where $\mu_{x}$ and $\sigma_{x}$ are defined in (19) and (20), respectively. After some algebra, we reach the conclusion that $\Pi\left(RB(\mu|x)\leq RB(\mu_{1}|x)|x\right)$ coincides with (9), but here $\sigma^{2}$ is random as defined in (17).

As for the KL approach, we compute $d(p^{prior},Q)$ and $d(p^{post},Q)$ as given respectively in (11) and (12). The approach makes a comparison between the concentrations of the prior and posterior distributions of the KL divergence via the relative belief ratio.

4.2 Elicitation of the prior

To elicit the prior, we consider the approach developed by Evan (2015, p.171). Suppose that it is known with virtual certainty (probability = 0.999) that $\mu\in(a,b)$ for specified values $a\leq b$. This is chosen to be as short as possible, based on the knowledge of the basic measurements being taken and without being unrealistic. We set $\mu_{0}=(a+b)/2$ (i.e, mid-point). With this choice, one hyper-parameter has been specified. It follows that

[TABLE]

This implies that

[TABLE]

An interval that contains virtually all the actual data measurements is given by $\mu\pm\sigma\Phi^{-1}(0.9995)$. Since this interval cannot be unrealistically too short or too long, we let $s_{1}$ and $s_{2}$ be the upper and lower bounds on the half-length of the interval so that

[TABLE]

That is,

[TABLE]

Now, from (25) and (26), we have:

[TABLE]

which determine the conditional prior for $\mu$. Note that $\lambda_{0}$ can be made bigger by choosing a bigger value of $b-a$.

Lastly, to obtain relevant values of $\alpha_{0}$ and $\beta_{0}$, let $G(\alpha_{0},\beta_{0},x)$ denotes the CDF of $\text{gamma}_{rate}(\alpha_{0},\beta_{0})$ distribution. From(26),

[TABLE]

Now, suppose we want to determine the lower and upper bounds in (27), so that this interval contains $1/\sigma^{2}$ with virtual certainty. Thus,

[TABLE]

Then we numerically solve (28) and (29) for $(\alpha_{0},\beta_{0})$.

4.3 Checking for Prior-data Conflict

To assess whether $(\bar{x}_{0},s_{0}^{2})$ is a reasonable value, we compute:

[TABLE]

where $T$, $M_{T}$ and $m_{T}$ are as defined in Section 4.1. Clearly, computing (30) should be done by simulation. Thus, for specified values of $\mu_{0},\lambda_{0}^{2},(\alpha_{0},\beta_{0})$, we generate $(\mu,\sigma^{2})$ as given in (17) and (18). Then generate $(\bar{x},s^{2})$ from the joint distribution given $(\mu,\sigma^{2})$ and evaluate $m_{T}(\bar{x},s^{2})$ using (22). Repeating this many times and recording the proportion of values of $m_{T}(\bar{x},s^{2})$ that are less than or equal to $m_{T}(\bar{x}_{0},s_{0}^{2})$ gives a Monte Carlo estimate of (30).

4.4 Checking for Bias

As in Section 3.4, the bias against the hypothesis $H_{0}:\mu=\mu_{1}$ is measured by computing (3) with $\psi_{0}=\mu_{1}$ and $RB(\mu|x)$ as given in (23). On the other hand, the bias in favor of the hypothesis $H_{0}:\mu=\mu_{1}$ is measured by computing (4) with $\psi_{0}=\mu_{1}$ and $RB(\mu_{1}|x)$ as defined in (23). The interpretation of the bias was given in Section 2.

4.5 The Algorithm

The following algorithm outlines the KL approach described in Section 4.

Algorithm B (New $t-$Test)

1. (i)

Elicit the hyperparameters $\mu_{0}$, $\lambda_{0}$ and $(\alpha_{0},\beta_{0})$ as described in Section 4.2. 2. (ii)

Generate $\mu$ and $\sigma^{2}$ as described in (17) and (18). 3. (iii)

Compute the KL distance between $P^{\text{Prior}}=N(\mu,\sigma^{2})$ and $Q=N(\mu_{1},\sigma^{2})$ as described in (11). Denote this distance by $D$. 4. (iv)

Repeat steps (ii) and (iii) to obtain a sample of $r_{1}$ values of $D$. 5. (v)

Generate $\mu_{x}$ and $\sigma^{2}_{x}$ from (19) and (20), respectively. 6. (vi)

Compute the KL distance between $P^{\text{Post}}=N(\mu,\sigma^{2})$ and $Q=N(\mu_{1},\sigma^{2})$ as described in (12). Denote this distance by $D_{x}$. 7. (vii)

Repeat steps (v) and (vi) to obtain a sample of $r_{2}$ values of $D_{x}$. 8. (viii)

Compute the relative belief ratio and the strength described in Algorithm A.

Note that, like Proposition 1, the approach in this case (i.e., when $\sigma^{2}$ is unknown) is also consistent as the sample size increases.

5 Examples

In this section, we consider three examples. The first one deals with a study on dental anxiety in adults, where the goal is to gauge the fear of adults of going to a dentist (McClave and Sincich, 2017, p. 398). For this, a random sample of 15 adults completed the Modified Dental Anxiety Scale questionnaire, where scores range from zero (no anxiety) to 25 (extreme anxiety). The sample mean score was 10.7 and the sample standard deviation was 3.6. We want to determine whether the mean Dental Anxiety Scale score for the population differs from 11. To construct the prior, we implement the elicitation algorithm described in Section 4.1 with $a=0$, $b=25$, $s_{1}=2$, $s_{2}=15$ and $\gamma=0.999$. Consequently, we have $\mu_{0}=12.5$, $\lambda_{0}=0.83$, $\alpha_{0}=1.29$ and $\beta_{0}=12.36$. To check if there is a prior-data conflict, (30) is computed to be 0.46, and that implies an indication of no prior-data conflict. The bias is also assessed by computing (3) with $\psi_{0}=11$. In this case the bias against the null hypothesis is 0.5136. On the other hand, the bias in favor of the null hypothesis is measured by computing (4) with $\psi_{0}=11\pm 0.5$, which gives 0.5192 (for $\psi_{0}=11.5$) and 0.5208 (for $\psi_{0}=10.5$). This shows equal bias either for or against the null hypothesis for this choice of prior. The value of the relative believe ratio test (distance and direct) with strength, the test of Rouder et. al. (2009) and the standard $t$-test are summarized in Table 1. It follows from the table that the null hypothesis is accepted by the three Bayesian tests, while it is not rejected by the $t$-test.

The second example considers an application about the age at which children start walking (Mann, 2016). A psychologist claims that the mean age at which children start walking is 12.5 months. To test this claim, she took a random sample of 18 children and found that the mean age at which they started walking was 12.9 with a standard deviation of 0.80 month. As in the previous example, the prior is constructed by setting $a=8$, $b=24$, $s_{1}=4$, $s_{2}=10$ and $\gamma=0.999$ the algorithm described in Section 4.1. It follows that $\mu_{0}=16$, $\lambda_{0}=0.8$, $\alpha_{0}=4.01$ and $\beta_{0}=329.78$. With this prior, it is found that (30)=1, which is a clear indication of no prior-data conflict. The bias is also assessed by computing (3) with $\psi_{0}=12.5$. In this case the bias against the null hypothesis is 0.4991. Moreover, the bias in favor of the null hypothesis is measured by computing (4) with $\psi_{0}=12.5\pm 0.5$, which gives 0.5169 (for $\psi_{0}=13$) and 0.5140 (for $\psi_{0}=12.0$). This shows equal bias either for or against the null hypothesis for this choice of prior. The results are reported in Table 2. Thus, the tests based on the relative belief ratio accept the null hypotheses while the other two tests reject the null hypothesis.

The last example deals with sugar production (Bluman, 2012, p. 457), where sugar is packed in 5-pound bags. An inspector suspects the bags may not contain 5 pounds. A sample of 50 bags produces a mean of 4.6 pounds and a standard deviation of 0.7 pound. To goal is to test if bags do not contain 5 pounds as stated. In the algorithm given in Section 4.1, we set $a=4$, $b=6$, $s_{1}=2$, $s_{2}=5$ and $\gamma=0.999$. We get $\mu_{0}=5$, $\lambda_{0}=0.2$, $\alpha_{0}=4.0077$ and $\beta_{0}=20.6106$. For this prior, (30)=0.6262, which means no prior-data conflict is found. The bias is evaluated by computing (3) with $\psi_{0}=4.6$. In this case, the bias against the null hypothesis is 0.4877. Additionally, the bias in favor of the null hypothesis is measured by computing (4) with $\psi_{0}=4.6\pm 0.5$, which gives 0.4876 (for $\psi_{0}=5.1$) and 0.4864 (for $\psi_{0}=4.1$). This demonstrate equal bias either for or against the null hypothesis for this choice for prior. The results are reported in Table 3. From the table, we see that all the previous tests reject the null hypothesis.

6 Concluding Remarks

A Bayesian approach to the standard one-sample $z$- and $t$- tests has been developed. The prior has been created through an elicitation algorithm. Then the prior is evaluated for the existence of prior-data conflict and bias. The use of the approach has been illustrated through several examples, in which it shows excellent performance.

The approach can be extended in several directions. For instance, it can be used to test the difference between two population means. Also, it can be modified to be a Bayesian alternative to the Hotelling’s $T^{2}$ test for the multivariate mean.