A Short Note on Almost Sure Convergence of Bayes Factors in the General Set-Up

TL;DR

This paper establishes the almost sure convergence of Bayes factors in a very general setting, including dependent data and misspecified models, extending previous asymptotic results.

Contribution

It provides the first comprehensive almost sure convergence theory for Bayes factors in a broad, non-i.i.d. framework, leveraging Shalizi's result.

Findings

Almost sure convergence of Bayes factors in general set-ups

Applicable to dependent data and misspecified models

Simplifies proof using Shalizi's result

Abstract

Although there is a significant literature on the asymptotic theory of Bayes factor, the set-ups considered are usually specialized and often involves independent and identically distributed data. Even in such specialized cases, mostly weak consistency results are available. In this article, for the first time ever, we derive the almost sure convergence theory of Bayes factor in the general set-up that includes even dependent data and misspecified models. Somewhat surprisingly, the key to the proof of such a general theory is a simple application of a result of Shalizi (2009) to a well-known identity satisfied by the Bayes factor.

Equations

$$B_n^{(12)}=\frac{m({\mathbf{X}}_n|{\mathcal{M}}_1)}{m({\mathbf{X}}_n|{\mathcal{M}}_2)}.$$

$$h(\theta )$$

$$h\left(A\right)$$

$$J(\theta )$$

$$J(A)$$

$$\underset{\theta \in A}{\text{mbox}essinf}g(\theta )=\sup \left\{r\in \mathbb{R}:g(\theta )>r,\text{mbox}foralmostall\theta \in A\right\},$$

$$\underset{n\to \infty }{\lim }\frac{1}{n}\log \left[\pi (\theta |{\mathbf{X}}_n)\right]=-J(\theta ),$$

$$\log \left[m({\mathbf{X}}_n|{\mathcal{M}}_i)\right]=\log \left[L_n({\theta }_i|{\mathcal{M}}_i)\right]+\log \left[\pi ({\theta }_i|{\mathcal{M}}_i)\right]-\log \left[\pi ({\theta }_i|{\mathbf{X}}_n,{\mathcal{M}}_i)\right].$$

$$\log \left[B_n^{(12)}\right]$$

$$\frac{1}{n}\log \left[B_n^{(12)}\right]$$

$$+\frac{1}{n}\log \left[\pi ({\theta }_1|{\mathcal{M}}_1)\right]-\frac{1}{n}\log \left[\pi ({\theta }_2|{\mathcal{M}}_2)\right]$$

$$-\frac{1}{n}\log \left[\pi ({\theta }_1|{\mathbf{X}}_n,{\mathcal{M}}_1)\right]+\frac{1}{n}\log \left[\pi ({\theta }_2|{\mathbf{X}}_n,{\mathcal{M}}_2)\right],$$

$$\underset{n\to \infty }{\lim }\frac{1}{n}\log \left[R_n({\theta }_i|{\mathcal{M}}_i)\right]=-h_i({\theta }_i),$$

$$\underset{n\to \infty }{\lim }\frac{1}{n}\log \left[\pi ({\theta }_i|{\mathbf{X}}_n,{\mathcal{M}}_i)\right]=-J_i({\theta }_i),$$

$$\underset{n\to \infty }{\lim }\frac{1}{n}\log \left[B_n^{(12)}\right]=-\left[h_1({\Theta }_1)-h_2({\Theta }_2)\right],$$

$$R_n(\theta )=\frac{L_n(\theta )}{p_n}$$

$$\underset{n\to \infty }{\lim }\frac{1}{n}\log \left[R_n(\theta )\right]=-h(\theta ).$$

$$\pi \left({\mathcal{G}}_n\right)\ge 1-\alpha \exp \left(-\beta n\right),\text{mbox}forsome\alpha >0,\beta >2h(\Theta );$$

$$n^{-1}\log \left[{\int }_A{R}_n(\theta )\pi (\theta )d\theta \right]\le \delta +\underset{n\to \infty }{\mathrm{limsup}}{n}^{-1}\log \left[{\int }_A{R}_n(\theta )\pi (\theta )d\theta \right],$$

$$x_t={\rho }_0{x}_{t-1}+{ϵ}_t,t=1,2,\dots ,$$

$$\log R_T({\rho }_1)=\left(\frac{{\rho }_0-{\rho }_1}{{\sigma }_0^2}\right)\left[\left(\sum _{t=1}^T{x}_{t-1}^2\right)\left(\frac{{\rho }_0+{\rho }_1}{2}\right)-\sum _{t=1}^T{x}_t{x}_{t-1}\right].$$

$$Cov(x_{t+h},x_t)\sim \frac{{\sigma }_0^2{\rho }_0^h}{1-{\rho }_0^2};h\ge 0,$$

$$E\left[\log R_T({\rho }_1)\right]$$

$$\sim -\left({\rho }_1-{\rho }_0\right)\left[\frac{(T-1)\left({\rho }_1+{\rho }_0\right)}{2(1-{\rho }_0^2)}-\frac{(T-1){\rho }_0}{(1-{\rho }_0^2)}\right],$$

$$\frac{E\left[\log R_T({\rho }_1)\right]}{T}\to -\frac{{\left({\rho }_1-{\rho }_0\right)}^2}{2(1-{\rho }_0^2)},\text{mbox}asT\to \infty .$$

$$h_1({\rho }_1)=\frac{{\left({\rho }_1-{\rho }_0\right)}^2}{2(1-{\rho }_0^2)}.$$

$$\underset{|{\rho }_1|<1}{\sup }\left|\frac{\log R_T({\rho }_1)}{T}+h_1({\rho }_1)\right|$$

$$=\underset{|{\rho }_1|<1}{\sup }\left|\frac{{\rho }_1-{\rho }_0}{{\sigma }_0^2}\right|\times \left|\left(\frac{{\sum }_{t=1}^T{x}_{t-1}^2}{T}\right)\left(\frac{{\rho }_1+{\rho }_0}{2}\right)-\frac{{\sum }_{t=1}^T{x}_t{x}_{t-1}}{T}-\frac{{\sigma }_0^2\left({\rho }_1-{\rho }_0\right)}{2(1-{\rho }_0^2)}\right|$$

$$\le \underset{|{\rho }_1|\le 1}{\sup }\left|\frac{{\rho }_1-{\rho }_0}{{\sigma }_0^2}\right|\times \left|\left(\frac{{\sum }_{t=1}^T{x}_{t-1}^2}{T}\right)\left(\frac{{\rho }_1+{\rho }_0}{2}\right)-\frac{{\sum }_{t=1}^T{x}_t{x}_{t-1}}{T}-\frac{{\sigma }_0^2\left({\rho }_1-{\rho }_0\right)}{2(1-{\rho }_0^2)}\right|$$

$$=\left|\frac{{\hat{\rho }}_1-{\rho }_0}{{\sigma }_0^2}\right|\times \left|\left(\frac{{\sum }_{t=1}^T{x}_{t-1}^2}{T}\right)\left(\frac{{\hat{\rho }}_1+{\rho }_0}{2}\right)-\frac{{\sum }_{t=1}^T{x}_t{x}_{t-1}}{T}-\frac{{\sigma }_0^2\left({\hat{\rho }}_1-{\rho }_0\right)}{2(1-{\rho }_0^2)}\right|$$

$$\le \kappa \left|\left(\frac{{\sum }_{t=1}^T{x}_{t-1}^2}{T}\right)\left(\frac{{\hat{\rho }}_1+{\rho }_0}{2}\right)-\frac{{\sum }_{t=1}^T{x}_t{x}_{t-1}}{T}-\frac{{\sigma }_0^2\left({\hat{\rho }}_1-{\rho }_0\right)}{2(1-{\rho }_0^2)}\right|,$$

$$\frac{{\sum }_{t=1}^T{x}_{t-1}^2}{T}\to \frac{{\sigma }_0^2}{1-{\rho }_0^2},$$

$$x_t{x}_{t-1}={\rho }_0{x}_{t-1}^2+{ϵ}_t{x}_{t-1},$$

$$\frac{{\sum }_{t=1}^T{ϵ}_t{x}_{t-1}}{T}\to 0,$$

$$\frac{{\sum }_{t=1}^T{x}_t{x}_{t-1}}{T}\to \frac{{\sigma }_0^2{\rho }_0}{1-{\rho }_0^2},$$

Full text

Keywords:* Bayes factor convergence; Kullback-Leibler divergence; Posterior consistency.*

1 Introduction

Bayes factors are well-established in the Bayesian literature for the purpose of model comparison. Briefly, given data $\boldsymbol{X}_{n}=\{X_{1},X_{2},\ldots,X_{n}\}$, where $n$ is the sample size, consider the problem of comparing any two models $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ associated with parameter spaces $\Theta_{1}$ and $\Theta_{2}$, respectively. For $i=1,2$, let the likelihoods, priors and the marginal densities for the two models be $L_{n}(\theta_{i}|\mathcal{M}_{i})=f_{\theta_{i}}(\boldsymbol{X}_{n}|\mathcal{M}_{i})$, $\pi(\theta_{i}|\mathcal{M}_{i})$ and $m(\boldsymbol{X}_{n}|\mathcal{M}_{i})=\int_{\Theta_{i}}L_{n}(\theta_{i}|\mathcal{M}_{i})\pi(d\theta_{i}|\mathcal{M}_{i})$, respectively. Then the Bayes factor of model $\mathcal{M}_{1}$ against $\mathcal{M}_{2}$ is given by

[TABLE]

Thus, $B^{(12)}_{n}$ can be interpreted as the quantification of the evidence of model $\mathcal{M}_{1}$ against model $\mathcal{M}_{2}$, given data $\boldsymbol{X}_{n}$. A comprehensive account of Bayes factors is provided in ?.

The asymptotic study of Bayes factor involves investigation of limiting properties of $B^{(12)}_{n}$ as $n$ goes to infinity. In particular, it is essential to guarantee the consistency property that $B^{(12)}_{n}$ goes to infinity almost surely when $\mathcal{M}_{1}$ is the better model and to zero almost surely when $\mathcal{M}_{2}$ is the better model. It is also important to obtain the rate of convergence of the Bayes factor. In the case of independent and identically distributed ($iid$) data, a relevant result is provided in ? and ?. Such strong “almost sure” convergence results are rare however, even when the data are independent but not identically distributed. Recently, ? obtained a strong, general result when the data are independent but not identically distributed and applied it to time-varying covariate and drift function selection in the context of systems of stochastic differential equations (see also ? for further application of Bayes factor asymptotics in stochastic differential equations). The other existing works on Bayes factor asymptotics are problem specific and even in such particular set-ups strong consistency results are seldom available (but see, for example, ?, ?, ?). For a comprehensive review of Bayes factor consistency, see ?.

We are interested in more general frameworks where the data may be dependent and where the possible models are perhaps all misspecified. We are not aware of any existing work on Bayes factor asymptotics in this direction. However, posterior convergence has been addressed by ?, and indeed, Theorem 2 of ? combined with a well-known identity satisfied by Bayes factors, holds the key to an elegant almost sure convergence result for the Bayes factor. The result depends explicitly on the average Kullback-Leibler divergence between the competing and the true models, even in such a general set-up. Here it is important to emphasize that although ? is essentially a review paper, the authors demonstrate for the first time with a specific example of nested models that the identity satisfied by the Bayes factor may be exploited to prove weak consistency of the latter, and provide general discussion regarding “in probability” Bayes factor convergence assuming that the identity is satisfied by the Bayes factor.

The rest of this article is structured as follows. In Section 2, based on ? we describe the general setting for our Bayes factor investigation, and provide the result of ? on which our main result on Bayes factor hinges. In Section 3 we provide our results on Bayes factor convergence. We make concluding remarks in Section 4. Additional details are provided in the online supplement, whose sections have the prefix “S-” when referred to in this paper.

2 The general set-up for model comparison using Bayes factors

Following ?, let us consider a probability space $(\Omega,\mathcal{F},P)$, sequence of random variables $\{X_{1},X_{2},\ldots\}$ taking values in the measurable space $(\aleph,\mathcal{X})$, having infinite-dimensional distribution $P$. In other words, the distribution $P$ is an infinite-dimensional distribution since it is the joint distribution of infinitely many random variables corresponding to a valid stochastic process. As guaranteed by Kolmogorov’s consistency result (see, for example, ?, ?), all finite-dimensional distributions associated with $P$ can be obtained by marginalizing over the remaining (infinite number of) variables. The theoretical development requires no restrictive assumptions on $P$ such as it being a product measure, Markovian, or exchangeable, thus paving the way for great generality.

Let $\mathcal{F}_{n}=\sigma(\boldsymbol{X}_{n})$ denote the natural filtration, that is, the $\sigma$-algebra generated by $\boldsymbol{X}_{n}$. Also, let the distributions of the processes adapted to $\mathcal{F}_{n}$ be denoted by $F_{\theta}$, where $\theta$ takes values in a measurable space $(\Theta,\mathcal{T})$. Here $\theta$ denotes the hypothesized probability measure associated with the unknown distribution of $\{X_{1},X_{2},\ldots\}$ and $\Theta$ is the set of hypothesized probability measures. In other words, assuming that $\theta$ is the infinite-dimensional distribution of the stochastic process $\{X_{1},X_{2},\ldots\}$, $F_{\theta}$ denotes the $n$-dimensional marginal distribution associated with $\theta$; $n$ is suppressed for ease of notation. For parametric models, the probability measure $\theta$ corresponds to a probability density with respect to some dominating measure (such as Lebesgue or counting measure) and consists of finite number of parameters. For nonparametric models, $\theta$ is usually associated with an infinite number of parameters and may not have a density with respect to $\sigma$-finite measures.

As in ?, we assume that $P$ and all the $F_{\theta}$ are dominated by a common measure with densities $p$ and $f_{\theta}$, respectively. In ? and in our case, the assumption that $P\in\Theta$ is not required so that all possible models are allowed to be misspecified.

Given a prior $\pi$ on $\theta$, we assume that the posterior distributions $\pi(\cdot|\boldsymbol{X}_{n})$ are dominated by a common measure for all $n\geq 1$; abusing notation, we denote the density at $\theta$ by $\pi(\theta|\boldsymbol{X}_{n})$.

Let $L_{n}(\theta)=f_{\theta}(\boldsymbol{X}_{n})$ be the likelihood and $p_{n}=p(\boldsymbol{X}_{n})$ be the marginal density of $\boldsymbol{X}_{n}$ under the true model $P$. Then following the notation of ?, for $A\subseteq\Theta$, let

[TABLE]

where, for any function $g:\Theta\mapsto\mathbb{R}$, where $\mathbb{R}$ is the real line,

[TABLE]

is the essential infimum of $g$ over the set $A$. Here “almost all” is with respect to the prior distribution. In words, essential infimum is the greatest lower bound which holds with prior probability one. With the above notations, six assumptions are used to prove Theorem 2 of ?. We provide the six assumptions in Section S-1 of the supplement, which we refer to as (A1)–(A6). Below we furnish Theorem 2 of ? which shall play the key role for our purpose of deriving almost sure convergence of Bayes factors.

Theorem 1** (Theorem 2 of ?)**

Consider assumptions (A1)–(A6). Then for all $\theta$ such that $\pi(\theta)>0$,

[TABLE]

almost surely with respect to the true model $P$, where $J(\theta)$ is given by (2.3).

3 Convergence of Bayes factors

For the model comparison problem using Bayes factors, we now assume the likelihoods and the priors of all the competing models satisfy (A1)–(A6), in addition to satisfying that $P$ and all the $F_{\theta}$ for $\theta\in\Theta_{1}\cup\Theta_{2}$ have densities with respect to a common $\sigma$-finite measure. We also assume that for $i=1,2$, the posterior $\pi(\cdot|\boldsymbol{X}_{n},\mathcal{M}_{i})$ associated with model $\mathcal{M}_{i}$ is dominated by the prior $\pi(\cdot|\mathcal{M}_{i})$, which is again absolutely continuous with respect to some appropriate $\sigma$-finite measure. These latter assumptions ensure that up to the normalizing constant, the posterior density associated with $\mathcal{M}_{i}$ is factorizable into the prior density times the likelihood. Indeed, for any $\theta_{i}\in\Theta_{i}$,

[TABLE]

Hence, the logarithm of the Bayes factor is given, for any $\theta_{1}\in\Theta_{1}$ and $\theta_{2}\in\Theta_{2}$, by (see, for example, ?, ?)

[TABLE]

so that

[TABLE]

where, for $i=1,2$, $R_{n}(\theta_{i}|\mathcal{M}_{i})=\frac{L_{n}(\theta_{i}|\mathcal{M}_{i})}{p_{n}}$.

Now let $J_{i}(\theta_{i})=h_{i}(\theta_{i})-h_{i}(\Theta_{i})$, where $h_{i}(\theta_{i})$ is defined as in (2.1) with $L_{n}(\theta)$ replaced with $L_{n}(\theta_{i}|\mathcal{M}_{i})$, and $h_{i}\left(A\right)=\underset{\theta_{i}\in A_{i}}{\mbox{ess~{}inf}}~{}h_{i}(\theta_{i})$, for any $A_{i}\subseteq\Theta_{i}$. Assumption (A3) then yields

[TABLE]

almost surely, and assuming that both the models and their associated priors satisfy assumptions (A1)–(A6), it follows using Theorem 1 that for $i=1,2$,

[TABLE]

almost surely.

Assuming that for $i=1,2$, $\pi(\theta_{i}|\mathcal{M}_{i})>0$ for all $\theta_{i}\in\Theta_{i}$, note that $\frac{1}{n}\log\left[\pi(\theta_{i}|\mathcal{M}_{i})\right]\rightarrow 0$ as $n\rightarrow\infty$, so that it follows using (3.2), (3.3) and (3.4), that

[TABLE]

almost surely with respect to $P$. We formalize this main result in the form of the following theorem:

Theorem 2** (Bayes factor convergence)**

Assume that for $i=1,2$, the competing models $\mathcal{M}_{i}$ satisfy assumptions (A1)–(A6), with parameter spaces $\Theta_{i}$, in addition to satisfying that $P$ and all the $F_{\theta}$ for $\theta\in\Theta_{1}\cup\Theta_{2}$ have densities with respect to a common $\sigma$-finite measure. We also assume that the posterior associated with $\mathcal{M}_{i}$ is dominated by the prior, which is again absolutely continuous with respect to some appropriate $\sigma$-finite measure, and that the priors satisfy $\pi(\theta_{i}|\mathcal{M}_{i})>0$ for all $\theta_{i}\in\Theta_{i}$. Then (3.5) holds almost surely with respect to the true infinite-dimensional probability measure $P$.

Since assumption (A3) is used directly for convergence of the likelihood ratios, it is perhaps desirable to consider sufficient conditions that ensure (A3). Such sufficient conditions, as noted in ?, can be found in ? and ?. Necessary and sufficient conditions for (A3) to hold has more recently been established in (?). However, in our experience, (A3) is usually easy to verify; see Section S-2 of the supplement; see also ?.

Theorem 2 provides an elegant convergence result for Bayes factors, explicitly in terms of differences between average Kullback-Leibler divergences between the competing and the true models. That such a result holds in the general set-up that includes even dependent data and misspecified models, is very encouraging. Indeed, we are not aware of any such result in the general set-up, although in the $iid$ situation ? and ? prove strong convergence of Bayes factor in terms of Kullback-Leibler divergences, taking misspecification into account. Theorem 2 readily leads to the following corollaries.

Corollary 3** (Consistency of Bayes factor)**

Without loss of generality, let $\mathcal{M}_{1}$ be the correct model and $\mathcal{M}_{2}$ be incorrect. Then $L_{n}(\theta_{1}|\mathcal{M}_{1})=p_{n}$ for all $\theta_{1}\in\Theta_{1}$, so that $h_{1}(\theta_{1})=0$ for all $\theta_{1}\in\Theta_{1}$, implying that $h_{1}(\Theta_{1})=0$. On the other hand, $h_{2}(\Theta_{2})>0$, so that by Theorem 2, $\underset{n\rightarrow\infty}{\lim}~{}\frac{1}{n}\log\left[B^{(12)}_{n}\right]=h_{2}(\Theta_{2})$. In other words, $B^{(12)}_{n}\rightarrow\infty$ exponentially fast, confirming consistency of the Bayes factor. If $\mathcal{M}_{1}$ is not necessarily the correct model but is a better model than $\mathcal{M}_{2}$ in the sense that its average Kullback-Leibler divergence $h_{1}(\Theta_{1})$ is smaller than $h_{2}(\Theta_{2})$, then again $B^{(12)}_{n}\rightarrow\infty$ exponentially fast, guaranteeing consistency.

Corollary 4** (Selection among a countable class of models)**

*Theorem 2 and Corollary 3 make it explicit that if the class of competing models is countable and contains the true model, it is selected by the Bayes factor, otherwise Bayes factor selects the model for which the average Kullback-Leibler divergence from the true model is minimized among the (countable) class of misspecified models, provided that the infimum is attained by some model. *

Corollary 5** (The case when two or more models are asymptotically correct)**

For simplicity let us consider two models $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ as before with parameter spaces $\Theta_{1}$ and $\Theta_{2}$ respectively. From Theorem 2 it follows that $\frac{1}{n}\log\left[B^{(12)}_{n}\right]\rightarrow 0$ almost surely if and only if $h_{1}(\Theta_{1})=h_{2}(\Theta_{2})$, that is, if and only if both the models are asymptotically correct in the average Kullback-Leibler sense. Note that the zero limit of $\frac{1}{n}\log\left[B^{(12)}_{n}\right]$ is the only logical limit here since any non-zero limit would lead the Bayes factor to lend infinitely more support to one model compared to the other even though both the competing models are correct asymptotically. The situation of zero limit of $\frac{1}{n}\log\left[B^{(12)}_{n}\right]$ may arise in the case of comparisons between nested models or when testing parametric versus nonparametric models. In these cases even though both the competing models are correct asymptotically, one may be a much larger model. For reasons of parsimony it then makes sense to choose the model with smaller dimensionality. If both the models are infinite-dimensional, for example, when comparing two sets of basis functions, then model combination seems to be the right step.

In Section S-2 of the supplement we illustrate Theorem 2 with an example with autoregressive models of the first order ($AR(1)$ models) comparing (asymptotically) stationary versus nonstationary models when the true model is (asymptotically) stationary. We show that asymptotically the Bayes factor heavily favours the (asymptotically) stationary model.

In Corollary 5, we have referred to comparisons with nonparametric models. However, recall that the results of Shalizi require the true model $P$ and all the postulated models $F_{\theta}$ to have densities with respect to a common dominating measure, and also the posteriors $\pi(\cdot|\boldsymbol{X}_{n})$ to be dominated by a common reference measure for all $n\geq 1$. These conditions are typically satisfied by parametric models, but not necessarily by nonparametric models. Indeed, in the case of the traditional nonparametric Bayesian analysis using the Dirichlet process prior, there is no parametric form of the likelihood as there is no density of the data $\boldsymbol{X}_{n}$ under this nonparametric set-up. Also, the prior is not dominated by any $\sigma$-finite measure, and so does not have any density. In other words, not all nonparametric models lead to posteriors that can be factorized as proportional to prior times likelihood, as our Bayes factor treatment requires. However, as we clarify in Section S-3 of the supplement with a series of various examples of nonparametric Bayesian set-ups, in general the aforementioned factorization of the posterior holds in Bayesian nonparametrics and the domination requirements of Shalizi also hold in general. However, we emphasize that we did not yet verify assumptions (A1)–(A6) for these cases, as we reserve this task for our future paper to be communicated elsewhere.

4 Conclusion

In this article, we have obtained an elegant almost sure convergence result for Bayes factors in the general set-up where the data may be dependent and where all possible models are allowed to be misspecified. To our knowledge, this is a first-time effort in this direction. Interestingly, in spite of the importance of the result, it follows rather trivially from Shalizi’s result on posterior consistency applied to the identity satisfied by Bayes factors. We assert that although similar results can be shown to hold in simpler set-ups (see ? and ? for the $iid$ set-up and ? for the independent and non-identical set-up) and perhaps under specific models, our contribution is a proof of a strong convergence result under a very general set-up that has not been considered before.

The generality of our result will enable Bayes factor based asymptotic comparisons of various models in various set-ups, for example, $k$-th order Markov models, hidden Markov models, spatial Markov random field models, models based on dependent systems of stochastic differential equations, parametric versus nonparametric models in the dependent data setting (? consider the $iid$ set-up and study “in-probability” convergence of Bayes factor comparing specific finite and infinite-dimensional models). dependent versus independent model set-ups, to name only a few. Moreover, even in the $iid$ data contexts, the existing Bayes factor asymptotic results for the specific problems are usually not directly based on Kullback-Leibler divergence. Since our result directly make use of Kullback-Leibler divergence in any set-up, it is much more appealing from this perspective compared to the existing results.

In our future endeavors, we shall explore the effectiveness of our result in various specific set-ups, along with comparisons with existing results whenever applicable.

Acknowledgment

We are sincerely grateful to the Editor-in-Chief, the Associate Editor and the referee whose detailed constructive comments have led to significant improvement of our manuscript.

Supplementary Material

S-1 Assumptions of Shalizi in the context of posterior consistency

• (A1)

Assume that the following likelihood ratio

[TABLE]

is $\mathcal{F}_{n}\times\mathcal{T}$-measurable for all $n\geq 1$.

• (A2)

For every $\theta\in\Theta$, the Kullback-Leibler divergence rate $h(\theta)$ exists (possibly being infinite) and is $\mathcal{T}$-measurable. Note that in the $iid$ set-up, $h(\theta)$ reduces to the Kullback-Leibler divergence between the true and the hypothesized model, so that $h(\theta)$ may be regarded as a generalized Kullback-Leibler divergence measure.

• (A3)

For each $\theta\in\Theta$, the generalized or relative asymptotic equipartition property holds, and so, almost surely with respect to $P$,

[TABLE]

Roughly, the terminology “asymptotic equipartition” refers to dividing up $\log\left[R_{n}(\theta)\right]$ into $n$ factors for large $n$ such that all the factors are asymptotically equal. Again, considering the $iid$ scenario helps clarify this point, as in this case each factor converges to the same Kullback-Leibler divergence between the true and the postulated model. With this understanding note that the purpose of condition (A3) is to ensure that relative to the true distribution, the likelihood of each $\theta$ decreases to zero exponentially fast, with rate being the Kullback-Leibler divergence rate (S-1.2).

• (A4)

Let $I=\left\{\theta:h(\theta)=\infty\right\}$. The prior $\pi$ on $\theta$ satisfies $\pi(I)<1$. Failure of this assumption entails extreme misspecification of almost all the hypothesized models $f_{\theta}$ relative to the true model $p$. With such extreme misspecification, posterior consistency is not expected to hold; see ? for details.

• (A5)

There exists a sequence of sets $\mathcal{G}_{n}\rightarrow\Theta$ as $n\rightarrow\infty$ such that:

1. (1)

$h\left(\mathcal{G}_{n}\right)\rightarrow h\left(\Theta\right)$, as $n\rightarrow\infty$. 2. (2)

[TABLE] 3. (3)

The convergence in (A3) is uniform in $\theta$ over $\mathcal{G}_{n}\setminus I$.

The sets $\mathcal{G}_{n}$ can be loosely interpreted as the sieves corresponding to the method of sieves (?) such that the behaviour of the likelihood ratio and the posterior on the sets $\mathcal{G}_{n}$ essentially carries over to $\Theta$. This can be anticipated from the first and the second parts of the assumption; the second part ensuring in particular that the parts of $\Theta$ on which the log likelihood ratio may be ill-behaved have exponentially small prior probabilities. The third part is more of a technical condition that is useful in proving posterior convergence through the sets $\mathcal{G}_{n}$. For further details, see ?.

For each measurable $A\subseteq\Theta$, for every $\delta>0$, there exists a random natural number $\tau(A,\delta)$ such that

[TABLE]

for all $n>\tau(A,\delta)$, provided $\underset{n\rightarrow\infty}{\lim\sup}~{}n^{-1}\log\left[\int_{A}R_{n}(\theta)\pi(\theta)d\theta\right]<\infty$. Regarding this, the following assumption has been made by Shalizi:

• (A6)

The sets $\mathcal{G}_{n}$ of (A5) can be chosen such that for every $\delta>0$, the inequality $n>\tau(\mathcal{G}_{n},\delta)$ holds almost surely for all sufficiently large $n$.

To understand the essence of this assumption, note that for almost every data set $\{X_{1},X_{2},\ldots\}$ there exists $\tau(\mathcal{G}_{n},\delta)$ such that (S-1.4) holds with $A$ replaced by $\mathcal{G}_{n}$ for all $n>\tau(\mathcal{G}_{n},\delta)$. Since $\mathcal{G}_{n}$ are sets with large enough prior probabilities, the assumption formalizes our expectation that $R_{n}(\theta)$ decays fast enough on $\mathcal{G}_{n}$. See ? for more detailed explanation.

S-2 Illustration of our result on Bayes factor with competing $AR(1)$ models

Let the true model $P$ stand for the following $AR(1)$ model:

[TABLE]

where $x_{0}\equiv 0$, $|\rho_{0}|<1$ and $\epsilon_{t}\stackrel{{\scriptstyle iid}}{{\sim}}N(0,\sigma^{2}_{0})$, for $t=1,2,\ldots$. We assume the competing models $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ to be of the same form as (S-2.1) but with the true parameter $\rho_{0}$ replaced with the unknown parameters $\rho_{1}$ and $\rho_{2}$, respectively, such that $|\rho_{1}|<1$ and $\rho_{2}\in(-1,1)^{c}\cap\mathbb{S}$, where $(-1,1)^{c}$ denotes complement of $(-1,1)$ and $\mathbb{S}$ is some compact set containing $[-1,1]$. For model $\mathcal{M}_{i}$; $i=1,2$, we assume that $x_{0}\equiv 0$ and $\epsilon_{t}\stackrel{{\scriptstyle iid}}{{\sim}}N(0,\sigma^{2}_{i})$; $t=1,2,\ldots$. For simplicity of illustration we assume for the time being that $\sigma_{1}$ and $\sigma_{2}$ are known, that is, $\sigma_{1}=\sigma_{2}=\sigma_{0}$, but see Section S-2.8 where we allow $\sigma_{1}$ and $\sigma_{2}$ to be unknown. Thus, we are interested in comparing (asymptotically) stationary and nonstationary $AR(1)$ models where the true $AR(1)$ model is (asymptotically) stationary. Note that $\Theta_{1}=(-1,1)$ and $\Theta_{2}=(-1,1)^{c}\cap\mathbb{S}$. We consider priors $\pi(\cdot|\mathcal{M}_{i})$; $i=1,2$, both of which have densities with respect to the Lebesgue measure. Let us first verify assumptions (A1)–(A6) with respect to $\mathcal{M}_{1}$. All the probabilities and expectations below are with respect to the true model $P$. Notationally, in this time series context we denote the sample size by the more natural notation $T$ rather than $n$.

S-2.1 Verification of (A1) for $\mathcal{M}_{1}$

Note that

[TABLE]

Thanks to continuity it is clear that $R_{T}(\rho_{1})$ is $\mathcal{F}_{T}\times\mathcal{T}$ measurable. In other words, (A1) holds.

S-2.2 Verification of (A2) for $\mathcal{M}_{1}$

It is easy to verify that under the true model $P$ the autocovariance function is given by

[TABLE]

where for any two sequences $\{a_{t}\}_{t=1}^{\infty}$ and $\{b_{t}\}_{t=1}^{\infty}$, $a_{t}\sim b_{t}$ stands for $a_{t}/b_{t}\rightarrow 1$ as $t\rightarrow\infty$. This leads to

[TABLE]

so that

[TABLE]

In other words, (A2) holds, with

[TABLE]

S-2.3 Verification of (A3) for $\mathcal{M}_{1}$

Rather than proving pointwise almost sure convergence of $\frac{\log R_{T}(\rho_{1})}{T}$ to $-h_{1}(\rho_{1})$, we prove the stronger result of almost sure uniform convergence in our example. Indeed, note that

[TABLE]

where step (S-2.5) follows due to compactness of $[-1,1]$; here $\hat{\rho}_{1}\in[-1,1]$ depends upon the data. In (S-2.6), $\kappa$ is a finite positive constant greater than the bounded positive quantity $\left|\frac{\hat{\rho}_{1}-\rho_{0}}{\sigma^{2}_{0}}\right|$.

Now observe that under $P$, the Markov chain $\left\{x_{t}:t=1,2,\ldots,\right\}$ is not only an asymptotically stationary process but is also irreducible and aperiodic (for definitions, see, for example, ? and ?). The latter two properties are easy to see because the chain can travel from any value in the real line to any set with positive Lebesgue measure in just one step with positive probability. Thus, the ergodic theorem holds, so that as $T\rightarrow\infty$,

[TABLE]

almost surely with respect to $P$. To deal with $\frac{\sum_{t=1}^{T}x_{t}x_{t-1}}{T}$, note that under $P$,

[TABLE]

and that $\left\{\epsilon_{t}x_{t-1}:t=2,3,\ldots\right\}$ is also an asymptotically stationary, irreducible and aperiodic Markov chain. Hence, applying ergodic theorem to the latter Markov chain, we obtain, using independence of $\epsilon_{t}$ and $x_{t-1}$ for all $t\geq 2$,

[TABLE]

as $T\rightarrow\infty$, almost surely with respect to $P$. It follows by combining (S-2.7), (S-2.8) and (S-2.9) that

[TABLE]

as $T\rightarrow\infty$, almost surely with respect to $P$. Applying (S-2.7) and (S-2.10) to (S-2.6) yields

[TABLE]

as $T\rightarrow\infty$, almost surely with respect to $P$. In other words, (A3) holds and the convergence is uniform.

S-2.4 Verification of (A4) for $\mathcal{M}_{1}$

In our example, (A4) holds trivially since $h_{1}(\rho_{1})=\frac{(\rho_{1}-\rho_{0})^{2}}{2(1-\rho^{2}_{0})}$, and $|\rho|<1$ almost surely. Specifically, $\pi(I|\mathcal{M}_{1})=0$.

S-2.5 Verification of (A5) for $\mathcal{M}_{1}$

First note that $h_{1}\left(\Theta_{1}\right)=\underset{\rho_{1}\in\Theta_{1}}{\mbox{ess~{}inf}}~{}h_{1}(\rho_{1})=\underset{\rho_{1}\in\Theta_{1}}{\mbox{ess~{}inf}}~{}\frac{(\rho_{1}-\rho_{0})^{2}}{2(1-\rho^{2}_{0})}=0$. Next, let $\mathcal{G}_{T}=\Theta_{1}$, for $T>0$. Then (A5) (1) and (A5) (2) hold trivially. Validation of (A5) (3) is exactly the same as our proof of uniform convergence of $\frac{\log R_{T}(\cdot)}{T}$ to $h_{1}(\cdot)$, provided in Section S-2.3. Hence, (A5) is satisfied.

S-2.6 Verification of (A6) for $\mathcal{M}_{1}$

Under (A1) – (A3), which we have already verified, it holds that (see equation (18) of ?) for any fixed $\mathcal{G}$ of the sequence $\mathcal{G}_{T}$, for any $\epsilon>0$ and for sufficiently large $T$,

[TABLE]

It follows that $\tau(\mathcal{G}_{T},\delta)$ is almost surely finite for all $T$ and $\delta$. We now argue that for sufficiently large $T$, $\tau(\mathcal{G}_{T},\delta)>T$ only finitely often with probability one. By equation (41) of ?,

[TABLE]

Since $\frac{1}{m}\log\int_{\mathcal{G}_{T}}R_{m}(\rho_{1})\pi(\rho_{1}|\mathcal{M}_{1})d\rho_{1}=\frac{1}{m}\log\int_{|\rho_{1}|\leq 1}R_{m}(\rho_{1})\pi(\rho_{1}|\mathcal{M}_{1})d\rho_{1}$, by the mean value theorem for integrals,

[TABLE]

for $\hat{\rho}_{T}\in[-1,1]$ depending upon the data.

Since $h_{1}(\mathcal{G}_{T})=h_{1}\left((-1,1)\right)=0$, and $h_{1}(\hat{\rho}_{T})\geq 0$, it follows from

[TABLE]

and (S-2.14) that

[TABLE]

Thus, it follows from (S-2.13), (S-2.6) and (S-2.8), that

[TABLE]

We first show that (S-2.15) is convergent. To simplify arguments, we first approximate $x_{t}=\sum_{k=1}^{t}\rho^{t-k}_{0}\epsilon_{k}$ by

[TABLE]

in the “in probability” sense. In $\tilde{x}_{t}$, $t_{0}$ is such that, for any given $\varepsilon>0$, for $t>t_{0}$,

[TABLE]

Since $\tilde{x}_{t}$ consists of only $t_{0}+1$ terms for any $t>t_{0}$, it is easier to handle compared to $x_{t}$, whose number of terms increases with $t$. Importantly, $\tilde{x}_{t}$ and $\tilde{x}_{t+t_{0}+k}$ are independent, for any $k\geq 1$. This property, which is not possessed by $x_{t}$, will be instrumental for making most of the terms zero associated with multinomial expansions required in our proceeding.

For the “in probability” fact, note that

[TABLE]

and

[TABLE]

due to (S-2.18). Since $\varepsilon>0$ is arbitrary, it follows that

[TABLE]

where $``\stackrel{{\scriptstyle P}}{{\longrightarrow}}"$ indicates “in probability” convergence. Now, $\left|x^{2}_{t}-\tilde{x}^{2}_{t}\right|=\left|x_{t}+\tilde{x}_{t}\right|\times\left|x_{t}-\tilde{x}_{t}\right|$, where $x_{t}$ is an irreducible, aperiodic Markov chain with mean zero Gaussian asymptotic stationary distribution with variance $\sigma^{2}_{0}/(1-\rho^{2}_{0})$, and $\tilde{x}_{t}$ is also asymptotically Gaussian with mean zero and variance $\sigma^{2}_{0}(1-\rho^{2(t_{0}+1)}_{0})/(1-\rho^{2}_{0})$. Hence, $\left|x_{t}+\tilde{x}_{t}\right|$ converges in probability to a finite random variable, and because of (S-2.21), it follows from the above representation that

[TABLE]

It then follows from the representation

[TABLE]

(S-2.22), and Theorem 7.15 of ? that

[TABLE]

Now note that for any finite integer $p\geq 1$,

[TABLE]

Noting that the multinomial expansion $(a_{1}+a_{2}+\cdots+a_{m})^{p}=\sum_{b_{1}+b_{2}+\cdots+b_{m}=p}\prod_{j=1}^{m}a^{b_{j}}_{j}$ (where $b_{1},\ldots,b_{m}$ are non-negative integers) consists of ${{m+p-1}\choose p}$ terms, it follows using asymptotic stationarity of $x_{t}$ and $\tilde{x}_{t}$ that both the expectations on the right hand side of (S-2.24) are of the order $O(1)$, as $m\rightarrow\infty$. Also, since for any finite $m$, the expectations are finite, it follows that the right hand side of (S-2.24) is finite, from which uniform integrability, and hence

[TABLE]

follows for integers $p\geq 1$. Hence, using binomial expansion, the Cauchy-Schwartz inequality and (S-2.25), it follows that

[TABLE]

so that

[TABLE]

In other words, for $p\geq 1$,

[TABLE]

Hence, while applying Markov’s inequality to the probability terms of the series (S-2.15), we can replace the moments associated with $x_{t}$ with those associated with $\tilde{x}_{t}$, for $m>T_{0}$, where $T_{0}$ is sufficienty large.

Now observe that

[TABLE]

For $m>T_{0}$, where $T_{0}$ is sufficiently large, $\left|\frac{\hat{\rho}_{T}-\rho_{0}}{2}\right|\times\left|\frac{\sum_{t=1}^{m}E\left(x^{2}_{t}\right)}{m}-\frac{\sigma^{2}_{0}}{1-\rho^{2}_{0}}\right|<\frac{\delta}{4\kappa}$, so that the (S-2.29) is exactly zero for $m>T_{0}$. Using Markov’s inequality for (S-2.29) where $m>T_{0}$ and replacing $x_{t}$ with $\tilde{x}_{t}$ in the right hand side of Markov’s inequality using (S-2.27) we obtain

[TABLE]

where $C$ is a positive constant. Now, $\left(\sum_{t=1}^{m}\left[\tilde{x}^{2}_{t-1}-E\left(\tilde{x}^{2}_{t-1}\right)\right]\right)^{5}$ admits the multinomial expansion of the form $(a_{1}+a_{2}+\cdots+a_{m})^{5}=\sum_{b_{1}+b_{2}+\cdots+b_{m}=5}\prod_{t=1}^{m}a^{b_{t}}_{t}$, where $a_{t}=\left[\tilde{x}^{2}_{t-1}-E\left(\tilde{x}^{2}_{t-1}\right)\right]$ and $b_{1},\ldots,b_{m}$ are non-negative integers. Observe that for any $t\geq 1$, $a_{t}$ and $a_{t+t_{0}+k}$ are independent for any $k\geq 1$, which enables factorization of $E\left(\prod_{t=1}^{m}a^{b_{t}}_{t}\right)$ into products of expectations of the independent terms. Since $E(a_{t})=0$ for $t=2,\ldots,m$, the expected product term becomes zero whenever it consists of at least one term of the form $E(a_{t})$, for any $t=2,\ldots,m$.

For the sake of convenience, let $m=(s+1)(t_{0}+1)$, where $s~{}(\geq 1)$ is an integer. Let $A_{l}=\left\{a_{t}:t=(l-1)t_{0}+1,\ldots,l(t_{0}+1)\right\}$, for $l=1,\ldots,(s+1)$. Then $A_{l}$ and $A_{l+2+r}$ are independent sets for any integer $l\geq 1$ and any integer $r\geq 0$.

When at least one $b_{t}=1$, the following argument gives an upper bound on the number of ways $E\left(\prod_{t=1}^{m}a^{b_{t}}_{t}\right)$ can be non-zero. Consider selecting 5 sets, say, $\left\{A_{l},A_{l+1},A_{l+2},A_{l+3},A_{l+4}\right\}$ from $\{A_{1},\ldots,A_{s+1}\}$, for some $l\geq 1$. Let $B_{l}=\left\{b_{t}:t=(l-1)t_{0}+1,\ldots,l(t_{0}+1)\right\}$ for $l=1,\ldots,(s+1)$, and consider setting one element of each of $B_{l+r}$; $r=1,\ldots,5$ to be 1 and the rest of the $b_{t}$’s to be zero. Then the number of such cases, namely, $O\left((s+1)\right)$ (since $t_{0}$ is a constant), provides an upper bound on the number of possible ways $E\left(\prod_{t=1}^{m}a^{b_{t}}_{t}\right)$ can be non-zero when at least one $b_{t}=1$.

Further cases of non-zero $E\left(\prod_{t=1}^{m}a^{b_{t}}_{t}\right)$ can occur when one of the $b_{t}$’s is $5$ and the rest are zeros, and when one of the $b_{t}$ is $3$, another is $2$, and the rest are zeros, so that there are $m+m(m-1)=m^{2}$ cases with respect to such choices.

Hence, in all there are $O\left(m^{2}\right)$ possible cases when $E\left(\prod_{t=1}^{m}a^{b_{t}}_{t}\right)$ is non-zero, and in the remaining cases $E\left(\prod_{t=1}^{m}a^{b_{t}}_{t}\right)=0$. In other words,

[TABLE]

since $\hat{\rho}_{T}\in[-1,1]$.

Now, (S-2.15) converges if and only if

[TABLE]

for sufficiently large $T_{0}$. Due to (S-2.28), (S-2.29) (which is exactly zero for $m>T_{0}$), (S-2.30) and (S-2.31), we see that (S-2.32) is dominated by some finite positive constant times the series

[TABLE]

The series (S-2.33) is convergent since it is bounded above by $\sum_{k=1}^{\infty}\frac{(T_{0}+k)}{(T_{0}+k)^{3}}\leq\sum_{k=1}^{\infty}\frac{1}{k^{2}}<\infty$.

Similar (and simpler) arguments and using the result

[TABLE]

shows that the series (S-2.16) also converges. Hence, (A6) stands verified.

Thus, (A1)–(A6) holds for $\mathcal{M}_{1}$.

S-2.7 Verification of Shalizi’s conditions for model $\mathcal{M}_{2}$

We now verify the same set of conditions for $\mathcal{M}_{2}$. As in $\mathcal{M}_{1}$, (A1) and (A2) easily hold; here $h_{2}(\rho_{2})=\frac{(\rho_{2}-\rho_{0})^{2}}{2(1-\rho^{2}_{0})}$ is of the same form as $h_{1}$. With respect to (A3) we verify pontwise convergence as required, rather than uniform convergence as in $\mathcal{M}_{1}$. Using (S-2.7), (S-2.8), (S-2.9) and (S-2.10), it is easily seen that $\frac{\log R_{T}(\rho_{2})}{T}+h_{2}(\rho_{2})\rightarrow 0$ almost surely, for all $\rho_{2}\in\Theta_{2}$. As in $\mathcal{M}_{1}$, it is clear that $\pi(I|\mathcal{M}_{2})=0$ so that (A4) holds.

As regards (A5), note that

[TABLE]

Now, in contrast with $\mathcal{M}_{1}$, here let $\mathcal{G}_{T}=\left\{\rho_{2}\in\Theta_{2}:|\rho_{2}|\leq\exp(\beta T)\right\}$, where $\beta>h_{2}(\Theta_{2})$, with $h_{2}(\Theta_{2})$ given by (S-2.34). It is easily seen that $\mathcal{G}_{T}\rightarrow\Theta_{2}$ and $h_{2}(\mathcal{G}_{T})\rightarrow h_{2}(\Theta_{2})$, as $T\rightarrow\infty$, so that (A5) (1) holds, and (A5) (2) is satisfied by Markov’s inequality. Since $\mathcal{G}_{T}$ and $\mathbb{S}$ are compact, verification of (A5) (3) follows in the same way as our proof of uniform convergence of $\frac{\log R_{T}(\cdot)}{T}$ to $-h_{1}(\cdot)$ in the case of $\mathcal{M}_{1}$, provided in Section S-2.3. That is, (A5) is satisfied for $\mathcal{M}_{2}$.

To verify (A6), first note that due to compactness of $\mathcal{G}_{T}$, the mean value theorem for integrals yields

[TABLE]

for some $\hat{\rho}_{T}\in\mathcal{G}_{T}$.

Since $h_{2}(\tilde{\rho}_{T})\geq h_{2}(\mathcal{G}_{T})$, it follows from

[TABLE]

and (S-2.35) that

[TABLE]

The rest of the validation of condition (A6) follows in the same way as in the case of $\mathcal{M}_{1}$, as detailed in Section S-2.6.

Hence, Theorem 2 of our main manuscript holds, so that

[TABLE]

that is, the Bayes factor heavily favours the (asymptotically) stationary model $\mathcal{M}_{1}$ over the nonstationary model $\mathcal{M}_{2}$. Since the true model $P$ is (asymptotically) stationary, this result is very encouraging.

S-2.8 Convergence of Bayes factor when $\rho_{1}$, $\rho_{2}$, $\sigma_{1}$ and $\sigma_{2}$ are all unknown

When apart from unknown $\rho_{1}$ and $\rho_{2}$, the error variances $\sigma^{2}_{1}$ and $\sigma^{2}_{2}$ associated with models $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ are also unknown, we consider the parameter spaces $\Theta_{1}=\left\{(\rho_{1},\sigma_{1}):|\rho_{1}|<1,\sigma_{1}\geq\eta\right\}$ and $\Theta_{2}=\left\{(\rho_{2},\sigma_{2}):\rho_{2}\in(-1,1)^{c}\cap\mathbb{S},\sigma_{2}\geq\eta\right\}$ associated with models $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$, respectively, where $0<\eta<\sigma_{0}$ is some small constant. For $i=1,2,$ we assume joint priors $\pi(\rho_{i},\sigma_{i}|\mathcal{M}_{i})$, having densities on $\Theta_{i}$, with respect to the Lebesgue measure. It can be easily seen that in this case, for $i=1,2$,

[TABLE]

Since $(1-\rho^{2}_{0})\log\frac{\sigma^{2}_{0}}{\sigma^{2}_{i}}+1\leq\log\frac{\sigma^{2}_{0}}{\sigma^{2}_{i}}+1\leq\frac{\sigma^{2}_{0}}{\sigma^{2}_{i}}$, (S-2.37) is non-negative. Also, as in the case with $\sigma_{1}=\sigma_{2}=\sigma_{0}$, it holds that $h_{1}(\Theta_{1})=0$ and $h_{2}\left(\Theta_{2}\right)=\min\left\{\frac{(1-\rho_{0})^{2}}{2(1-\rho^{2}_{0})},\frac{(1+\rho_{0})^{2}}{2(1-\rho^{2}_{0})}\right\}$. Further, note that $\pi(I|\mathcal{M}_{i})=0$, for $i=1,2$. Thus, conditions (A1)–(A4) are easily seen to hold for both the competing models.

We now verify the remaining conditions for the models. As regards $\mathcal{G}_{T}$, here we set

[TABLE]

for model $\mathcal{M}_{1}$ where $\beta>h_{1}(\Theta_{1})=0$, and for model $\mathcal{M}_{2}$ we set

[TABLE]

where $\beta>h_{2}(\Theta_{2})$. Note that there exists $T_{0}\geq 1$ such that $\sigma_{0}\leq\exp(\beta T)$ for $T\geq T_{0}$. Hence, $h_{1}(\mathcal{G}_{T})=h_{1}(\Theta_{1})=0$ and $h_{2}(\mathcal{G}_{T})=h_{2}(\Theta_{2})=\min\left\{\frac{(1-\rho_{0})^{2}}{2(1-\rho^{2}_{0})},\frac{(1+\rho_{0})^{2}}{2(1-\rho^{2}_{0})}\right\}$, for $T\geq T_{0}$. Hence, (A5) (1) holds for both $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$. Now observe that

[TABLE]

so that (A5) (2) holds for $\mathcal{M}_{1}$. For $\mathcal{M}_{2}$, denoting by $E_{2}$ the expectation with respect to $\pi(\cdot|\mathcal{M}_{2})$, note that

[TABLE]

where

[TABLE]

and

[TABLE]

by Markov’s inequality. It follows that

[TABLE]

that is, (A5) (2) holds for $\mathcal{M}_{2}$. That (A5) (3) holds for $\mathcal{M}_{1}$ can be shown in the same way as in Section S-2.3, by replacing $|\rho_{1}|<1$ by $|\rho_{1}|\leq 1$ in $\mathcal{G}_{T}$ and using the assumption that $\sigma_{1}\geq\eta>0$. For $\mathcal{M}_{2}$ as well, (A5) (3) can be seen to hold in the same way using compactness of $\mathcal{G}_{T}$ and $\mathbb{S}$, and the assumption that $\sigma_{2}$ is bounded away from zero.

Now observe that for model $\mathcal{M}_{1}$, since $h_{1}(\mathcal{G}_{T})=0$ for $T\geq T_{0}$, it can be shown in the same way as in Section S-2.6 that

[TABLE]

holds for $T\geq T_{0}$. The same holds for model $\mathcal{M}_{2}$ using compactness of $\mathcal{G}_{T}$, as shown in Section S-2.7 in the context of verification of (A6) for $\mathcal{M}_{2}$ when $\sigma_{2}=\sigma_{0}$. Finally observe that it is sufficient to establish convergence of $\sum_{T=T_{0}}^{\infty}P\left(\tau(\mathcal{G}_{T},\delta)>T\right)$ for large enough $T_{0}$, which can be done similarly as before, for both $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$.

Hence, Theorem 2 of our main manuscript is applicable to this situation and the result remains the same as (S-2.36).

S-3 A first look at the applicability of our Bayes factor result to some infinite-dimensional models

S-3.1 Traditional Dirichlet process model: undominated case

Theorem 2 requires the unnormalized posterior to admit factorization as the prior times the likelihood. It is well-known that for the original nonparametric models associated with the Dirichlet process prior (?) such factorization is not possible, since there is no parametric form of the likelihood. In other words, if $[X_{1},\ldots,X_{T}|F]\stackrel{{\scriptstyle iid}}{{\sim}}F$, where $F\sim DP\left(\alpha F_{0}\right)$, where $DP\left(\alpha F_{0}\right)$ stands for Dirichlet process with base measure $F_{0}$ and precision parameter $\alpha$, then the likelihood associated with the data $X_{1},\ldots,X_{T}$ does not have a parametric form, and although the posterior $\pi(F|\boldsymbol{X}_{T})$ is well-defined, it is not dominated by any $\sigma$-finite measure (see, for example, Proposition 7.7 of ?), and hence does not have a density. This of course prevents factorization of the posterior of $F$ as the prior times likelihood. Moreover, recall that ? also assumes the existence of a common reference measure for the posteriors $\pi(\cdot|\boldsymbol{X}_{T})$, for all $T$, which does not hold here. Indeed, such an assumption is valid in the usual dominated case of Bayes theorem where the aforementioned factorization is possible; in such (usually parametric) cases, the prior is the natural common dominating measure (see ?, for example).

S-3.2 Dirichlet process mixture model: dominated case

Since Dirichlet process supports discrete distributions with probability one, the modeling style described in Section S-3.1 is inappropriate if the data $\boldsymbol{X}_{T}$ arises from some continuous distribution. Hence, for such data it is usual in Bayesian nonparametrics based on the Dirichlet process prior to consider the following mixture model (see, for example, ?):

[TABLE]

where $f(\cdot|\xi)$ is some standard continuous density, usually Gaussian, given $\xi\sim F$, where $F\sim DP\left(\alpha F_{0}\right)$. By Sethuraman’s construction (?), $F(\cdot)=\sum_{i=1}^{\infty}p_{i}\delta_{\xi_{i}}(\cdot)$, with probability one, where, for $i=1,2,\ldots$, $\xi_{i}\stackrel{{\scriptstyle iid}}{{\sim}}F_{0}$, and for any $\xi$, $\delta_{\xi}(\cdot)$ denotes the point mass on $\xi$. Also, for $i=1,2,\ldots$, $p_{i}=V_{i}\prod_{j<i}(1-V_{j})$, where $V_{i}\stackrel{{\scriptstyle iid}}{{\sim}}Beta(\alpha,1)$. It is easy to verify that $\sum_{i=1}^{\infty}p_{i}=1$, almost surely. Application of Sethuraman’s construction in (S-3.1) yields the equivalent infinite mixture representation

[TABLE]

where $\theta=(\xi_{1},\xi_{2},\ldots,V_{1},V_{2},\ldots)$ is the infinite-dimensional parameter. The prior on $\theta$ is already specified by the $iid$ $F_{0}$ and $Beta(\alpha,1)$ distributions, and is the infinite product probability measure associated with these $iid$ distributions, so that each factor of the product of the probability measures is dominated by the Lebesgue measure. In this case, the posterior of $\theta$ admits the representation

[TABLE]

and hence the representation of Bayes factor in terms of the prior and the likelihood holds in this case, as required by Theorem 2 of our main manuscript. Moreover, the posterior $\pi(\cdot|\boldsymbol{X}_{T})$ is absolutely continuous with respect to $\pi(\cdot)$ for all $T$, as assumed by ?.

S-3.3 Polya urn based mixture obtained by integrating out random $F$: dominated case but $\mathcal{T}$ changes

with $T$

Assume that for $t=1,\ldots,T$, $[X_{t}|\phi_{t}]\sim f(\cdot\phi_{t})$, independently, and $\phi_{1},\ldots,\phi_{T}\stackrel{{\scriptstyle iid}}{{\sim}}F$, where $F\sim DP\left(\alpha F_{0}\right)$. This is equivalent to the Dirichlet process mixture model (S-3.1), but if $F$ is integrated out, then the joint distribution of $\phi_{1},\ldots,\phi_{T}$ is given by the Polya urn scheme, that is, $\phi_{1}\sim F_{0}$, and for $t=2,\ldots,T$, $[\phi_{t}|\phi_{1},\ldots,\phi_{t-1}]\sim\frac{\alpha F_{0}}{\alpha+t-1}+\frac{\sum_{j=1}^{t-1}\delta_{\phi_{j}}}{\alpha+t-1}$ (see, for example, ?, ?). The joint prior distribution of $\phi_{1},\ldots,\phi_{T}$ has a density with respect to a measure composed of Lebesgue measures in lower dimensions; see Lemma 1.99 of ? for the exact forms of the density and the dominating measure. Hence, in this case the posterior of $\phi_{1},\ldots,\phi_{T}$ is proportional to the prior times the likelihood, where the likelihood is given by $\prod_{t=1}^{T}f(X_{t}|\phi_{t})$, and the posterior is dominated by the prior probability measure. Hence, a countably infinite convex combination of the prior probability measures dominates the posterior of $\phi_{1},\ldots,\phi_{T}$ for all $T$, as required for the results of ? to hold. However, ? assumes that the $\sigma$-field $\mathcal{T}$ associated with the parameter space $\Theta$ does not change with $T$, which does not hold in this case.

S-3.4 Polya urn based finite mixture: dominated case and $\mathcal{T}$ remains fixed

? (see also ?, ?) introduce the following finite mixture model based on Dirichlet process:

[TABLE]

where $f(\cdot|\phi)$ is any standard density as before, given parameter(s) $\phi$, and $M~{}(>1)$ is some fixed integer. Integrating out $F$ yields the following Polya urn scheme for the joint distribution of $\phi_{1},\ldots,\phi_{M}$: $\phi_{1}\sim F_{0}$, and for $t=2,\ldots,M$, $[\phi_{t}|\phi_{1},\ldots,\phi_{t-1}]\sim\frac{\alpha F_{0}}{\alpha+t-1}+\frac{\sum_{j=1}^{t-1}\delta_{\phi_{j}}}{\alpha+t-1}$. Here $\theta=(\phi_{1},\ldots,\phi_{M})$, which is of fixed, finite size, even though the problem is induced by the nonparametric Dirichlet process prior. Also clearly the $\sigma$-field $\mathcal{T}$ associated with the parameter space $\Theta$ does not change with $T$. Thus, in this set-up, not only is the posterior written in terms of product of the prior and the likelihood, but is dominated by the Polya urn based prior of $\theta$, for all sample sizes $T$.

S-3.5 Nonparametric Bayesian using the Polya tree prior: dominated case

?, ? proposed the Polya tree prior for the random probability measure $F$ as an alternative to the Dirichlet process prior. Briefly, one starts with a partition $\pi_{1}=\{B_{0},B_{1}\}$ of the sample space $\Omega$, so that $\Omega=B_{0}\cup B_{1}$. This procedure is then continued with $B_{0}=B_{00}\cup B_{01}$, $B_{1}=B_{10}\cup B_{11}$, etc. At level $m$, the partition is then $\pi_{m}=\{B_{\epsilon}:\epsilon=\epsilon_{1}\ldots\epsilon_{m}\}$, where $\epsilon$ are all binary sequences of length $m$. Let $\Pi=\left\{\pi_{m}:m=1,2,\ldots\right\}$, and $\mathcal{A}=\left\{\alpha_{\epsilon}\right\}$ be a sequence of non-negative numbers, one for each partitioning subset. Now, if $Y_{\epsilon 0}=F(B_{\epsilon 0}|B_{\epsilon})\sim Beta(\alpha_{\epsilon 0},\alpha_{\epsilon 1})$ independently with respect to the $\epsilon$’s, then $F$ is said to have the Polya tree prior $PT(\Pi,\mathcal{A})$.

It can be shown that if $\alpha_{\epsilon}\propto m^{-1/2}$, the Polya tree prior reduces to the Dirichlet process prior, confirming that the latter is a special case of the Polya tree prior. However, the most important property of the Polya tree prior is that with appropriate choices of the $\alpha_{\epsilon}$, $F$ can be made absolutely continuous with respect to the Lebesgue measure. Specifically, if $\alpha_{\epsilon}\propto m^{2}$, for the $m$-th level subset, then $F$ is dominated by the Lebesgue measure almost surely. Hence, if $[X_{1},\ldots,X_{T}|F]\sim F$ and $F\sim PT(\Pi,\mathcal{A})$, with $\alpha_{\epsilon}\propto m^{2}$, then the likelihood is available almost surely. Here we may set $\theta=\left\{Y_{\epsilon 0}:\epsilon=\epsilon_{1}\ldots\epsilon_{m},m=1,2,\ldots\right\}$, which has the infinite product prior measure. The posterior of $F$ given $\boldsymbol{X}_{T}$, which is also a Polya tree process, is dominated by $\pi(\theta)$ for all $T>0$. Similar issues hold for the extended Polya tree prior, namely, the optional Polya tree prior proposed by ?.

S-3.6 Bayesian density estimation using the generalized lognormal process prior: dominated case

? model the unknown density $f(x)$ with respect to measure $\lambda$ as

[TABLE]

where $W$ is a generalized lognormal process over $\mathcal{X}$. The generalized lognormal process has distribution $\Lambda_{\eta}$ given by (see ?)

[TABLE]

where $-\infty<\eta<\infty$ and the expectations are taken with respect to the usual lognormal process, that is, with respect to $W=\exp\left(Z\right)$, where $Z$ is a Gaussian process. In (S-3.8), $\mathbb{I}_{A}$ is the indicator of the set $A$, where $A$ belongs to the Borel $\sigma$-field associated with the space of functions from $\mathcal{X}$ to $(0,\infty)$. The properties and moments of the lognormal process are provided in ?.

In this formulation, the likelihood with respect to $iid$ data $X_{1},\ldots,X_{T}$ is defined via (S-3.7). The prior distribution, as well as the posterior distribution of $\Theta=W$ for all $T\geq 1$, are absolutely continuous with respect to the distribution of the lognormal process $W=\exp\left(Z\right)$, where $Z$ is a Gaussian process.

S-3.7 Bayesian regression using Gaussian process: dominated case

Consider the following regression model with covariates $\left\{C_{t}:t=1,\ldots,T\right\}$ (see ?, for example):

[TABLE]

where $\varphi$ is a probability measure on the positive part of the real line, and in (S-3.9), $GP\left(\mu(\cdot),K(\cdot,\cdot)\right)$ stands for the Gaussian process with mean function $E\left[\zeta(c)\right]=\mu(c)$ for all $c\in\mathfrak{C}$, where $\mathfrak{C}$ is the space of covariates, and positive definite covariance function $Cov\left(\zeta(c_{1}),\zeta(c_{2})\right)=K(c_{1},c_{2})$, for all $c_{1},c_{2}\in\mathcal{C}$. Here, by positive definite function $K(\cdot,\cdot)$ on $\mathfrak{C}\times\mathfrak{C}$, we mean $\int K(c,c^{\prime})g(c)g(c^{\prime})d\nu(c)d\nu(c^{\prime})>0$ for all non-zero functions $g\in L_{2}\left(\mathfrak{C},\nu\right)$, where $L_{2}\left(\mathfrak{C},\nu\right)$ denotes the space of functions square-integrable on $\mathfrak{C}$ with respect to the measure $\nu$.

In what follows we borrow the statements of the following definition of eigenvalue and eigenfunction, and the subsequent statement of Mercer’s theorem from ?.

Definition 6

A function $\psi(\cdot)$ that obeys the integral equation

[TABLE]

is called an eigenfunction of the kernel $K$ with eigenvalue $\lambda$ with respect to the measure $\nu$.

We assume that the ordering is chosen such that $\lambda_{1}\geq\lambda_{2}\geq\cdots$. The eigenfunctions are orthogonal with respect to $\nu$ and can be chosen to be normalized so that $\int_{\mathfrak{C}}\psi_{i}(c)\psi_{j}(\mathbf{x})d\nu(c)=\delta_{ij}$, where $\delta_{ij}=1$ if $i=j$ and [math] otherwise.

The following well-known theorem (see, for example, ?) expresses the positive definite kernel $K$ in terms of its eigenvalues and eigenfunctions.

Theorem 7** (Mercer’s theorem)**

Let $(\mathfrak{C},\nu)$ be a finite measure space and $\mathcal{C}\in L_{\infty}\left(\mathfrak{C}^{2},\nu^{2}\right)$ be a positive definite kernel. By $L_{\infty}\left(\mathfrak{C}^{2},\nu^{2}\right)$ we mean the set of all measurable functions $K:\mathfrak{C}^{2}\mapsto\mathbb{R}$ which are essentially bounded, that is, bounded up to a set of $\nu^{2}$-measure zero. For any function $K$ in this set, its essential supremum, given by $\inf\left\{r\geq 0:|\mathcal{K}(c,c^{\prime})|<r,~{}\mbox{for almost all}~{}(c,c^{\prime})\in\mathfrak{C}\times\mathfrak{C}\right\}$ serves as the norm $\|K\|$.

Let $\psi_{j}\in L_{2}\left(\mathfrak{C},\nu\right)$ be the normalized eigenfunctions of $K$ associated with the eigenvalues $\lambda_{j}(K)>0$. Then

• (a)

the eigenvalues $\left\{\lambda_{j}(K)\right\}_{j=1}^{\infty}$ are absolutely summable.

• (b)

$K(c,c^{\prime})=\sum_{j=1}^{\infty}\lambda_{j}(K)\psi_{j}(c)\bar{\psi_{j}}(c^{\prime})$* holds $\nu^{2}$-almost everywhere, where the series converges absolutely and uniformly $\nu^{2}$-almost everywhere. In the above, $\bar{\psi_{j}}$ denotes the complex conjugate of $\psi_{j}$.*

It follows that the Gaussian process $\zeta$ admits the representation below almost surely:

[TABLE]

where, for $i=1,2,\ldots$, $e_{i}\stackrel{{\scriptstyle iid}}{{\sim}}N(0,1)$. The above representation for Gaussian processes is popularly known as the Karhunen-Loève expansion (see, for example, ?).

Hence, both the likelihood and the prior can be parameterized in terms of $\psi_{i}(\cdot);~{}i=1,2,\ldots$ and $\boldsymbol{\epsilon}=\left\{e_{i};~{}i=1,2,\ldots\right\}$, the latter being unknown and having the infinite product prior distribution such that $e_{i}\stackrel{{\scriptstyle iid}}{{\sim}}N(0,1)$; $i=1,2,\ldots$. Letting $\theta=(\boldsymbol{\epsilon},\sigma)$, note that the posterior $\pi(\theta|\boldsymbol{X}_{T})$, for all $T>0$, is clearly dominated by this infinite product prior measure times $\varphi$.