On posterior contraction of parameters and interpretability in Bayesian mixture modeling

TL;DR

This paper investigates how Bayesian mixture models' parameters converge post-data, comparing different priors and addressing model misspecification, with implications for interpretability and model complexity.

Contribution

It provides new theoretical results on posterior contraction rates under various priors and misspecification, highlighting the impact of kernel choices on interpretability.

Findings

Optimal contraction rates with explicit component priors.

Consistent recovery of the number of components with nonparametric priors.

Kernel choice significantly affects contraction under misspecification.

Abstract

We study posterior contraction behaviors for parameters of interest in the context of Bayesian mixture modeling, where the number of mixing components is unknown while the model itself may or may not be correctly specified. Two representative types of prior specification will be considered: one requires explicitly a prior distribution on the number of mixture components, while the other places a nonparametric prior on the space of mixing distributions. The former is shown to yield an optimal rate of posterior contraction on the model parameters under minimal conditions, while the latter can be utilized to consistently recover the unknown number of mixture components, with the help of a fast probabilistic post-processing procedure. We then turn the study of these Bayesian procedures to the realistic settings of model misspecification. It will be shown that the modeling choice of kernel density functions plays perhaps the most impactful roles in determining the posterior contraction rates in the misspecified situations. Drawing on concrete posterior contraction rates established in this paper we wish to highlight some aspects about the interesting tradeoffs between model expressiveness and interpretability that a statistical modeler must negotiate in the rich world of mixture modeling.

Equations

$$p_{G_0}:=\int f(x|\theta )d{G}_0(\theta )=\sum _{i=1}^{k_0}{p}_i^{0}f(x|{\theta }_i^0)$$

$$W_r(G,G^{\prime })$$

$$\displaystyle\sum_{|\kappa|=r}\biggr{|}\bigg{(}\frac{\partial^{|\kappa|}f}{\partial\theta^{\kappa}}(x|\theta_{1})-\frac{\partial^{|\kappa|}f}{\partial\theta^{\kappa}}(x|\theta_{2})\biggr{)}\gamma^{\kappa}\biggr{|}\leq C\|\theta_{1}-\theta_{2}\|^{\delta}\|\gamma\|$$

$$\sum _{l=0}^r\sum _{|\eta |=l}\sum _{i=1}^k{\alpha }_{\eta }^{(i)}\frac{{\partial }^{|\eta |}f}{\partial {\theta }^{\eta }}(x|{\theta }_i)=0$$

$$h(p_G,p_{G_0})\gtrsim W_1(G,G_0)$$

$$\displaystyle\Pi(G\in B\bigr{|}X_{1},\dots,X_{n})=\frac{\int_{B}\prod_{i=1}^{n}p_{G}(X_{i})\mathrm{d}\Pi(G)}{\int_{\overline{\mathcal{G}}(\Theta)}\prod_{i=1}^{n}p_{G}(X_{i})\mathrm{d}\Pi(G)}$$

$$\mathbf{p}=(p_1,\dots ,p_k)$$

$${\theta }_1,\dots ,{\theta }_k$$

$$X_1,\dots ,X_n|G=\sum _{i=1}^k{p}_i{\delta }_{{\theta }_i}$$

$$\displaystyle\Pi\biggr{(}G\in{\mathcal{E}_{k_{0}}}(\Theta):W_{1}(G,G_{0})\gtrsim(\log n/n)^{1/2}\biggr{|}X_{1},\ldots,X_{n}\biggr{)}\overset{p_{G_{0}}}{\longrightarrow}0.$$

$$\displaystyle\Pi\biggr{(}G\in{\mathcal{O}_{\overline{k}}}(\Theta):W_{2}(G,G_{0})\gtrsim(\log n/n)^{1/4}\biggr{|}X_{1},\ldots,X_{n}\biggr{)}\overset{p_{G_{0}}}{\longrightarrow}0.$$

$$\displaystyle\Pi\biggr{(}\exists I\subset\{1,\ldots,k\},|I|=k-k_{0}\ \text{s.t.}\ \sum_{i\in I}p_{i}<n^{-1/2+\epsilon}\biggr{|}X_{1},\ldots,X_{n}\biggr{)}\overset{p_{G_{0}}}{\longrightarrow}1.$$

$$K$$

$$\mathbf{p}=(p_1,\dots ,p_k)|K=k$$

$${\theta }_1,\dots ,{\theta }_k|K=k$$

$$X_1,\dots ,X_n|G=\sum _{i=1}^k{p}_i{\delta }_{{\theta }_i}$$

$$\displaystyle\Pi\biggr{(}G\in\overline{\mathcal{G}}(\Theta):W_{1}(G,G_{0})\lesssim(\log n/n)^{1/2}\biggr{|}X_{1},\ldots,X_{n}\biggr{)}\to 1$$

$$\displaystyle\Pi\biggr{(}G\in\overline{\mathcal{G}}(\Theta):W_{r}(G,G_{0})\leq\delta\omega_{n}\biggr{|}X_{1},\ldots,X_{n}\biggr{)}\overset{p_{G_{0}}}{\longrightarrow}1$$

$$G_{\ast }\in \underset{G\in \mathcal{P}(\Theta )}{\mathrm{argmin}}K(p_{G_0,f_0},p_G).$$

$${\mathcal{M}}^{\ast }:=\left\{G_{\ast }\in \mathcal{P}(\Theta ):G_{\ast }\in \underset{G\in \mathcal{P}(\Theta )}{\mathrm{argmin}}K(p_{G_0,f_0},p_G)\right\}.$$

$${\overline{h}}^2(p_{G_1},p_{G_2}):=\frac{1}{2}\int {\left(\sqrt{p_{G_1}(x)}-\sqrt{p_{G_2}(x)}\right)}^2\frac{p_{G_0,f_0}(x)}{p_{G_{\ast }}(x)}dx.$$

$$\Pi \left(W_r^r(G,G_{\ast })\le (2^r+1){ϵ}^r\right)$$

$$\gtrsim \frac{\gamma \Gamma (\gamma )D!q_D}{D}{\left(c_0{\left(\frac{ϵ}{\text{Diam}(\Theta )}\right)}^d\right)}^D{\left(\frac{1}{D}{\left(\frac{ϵ}{\text{Diam}(\Theta )}\right)}^r\right)}^{\gamma (D-1)/D}$$

$$\left\{f(\cdot |\theta ),\theta \in \Theta \subset {\mathbb{R}}^d:f(x|\theta ):=\frac{\exp (-(x-\theta {)}^{\top }{\Sigma }^{-1}(x-\theta )/2)}{|2\pi \Sigma {|}^{-1/2}}\right\},$$

$$\underset{x\in {\mathbb{R}}^d,\theta \in \Theta ,{\theta }_0\in \text{supp}(G_0)}{\sup }\frac{f(x|\theta )}{f_0(x|{\theta }_0)}\text{mathbbm}{1}_{\Vert x{\Vert }_2\le R}\le C_1\exp (C_0{R}^{\alpha }).$$

$$\displaystyle\overline{h}(p_{G},p_{G^{\prime}})\geq C\exp\biggr{(}-(1+8\lambda_{\max}(\lambda_{\min}^{-1}+C_{0}))/W_{2}^{2}(G,G^{\prime})\biggr{)}.$$

$$\displaystyle\Pi\biggr{(}G\in\overline{\mathcal{G}}(\Theta):W_{2}(G,G_{*})\lesssim\left(\frac{\log\log n}{\log n}\right)^{1/2}\biggr{|}X_{1},\ldots,X_{n}\biggr{)}\to 1$$

$$f(x|\theta )=\frac{2}{\lambda (2\pi {)}^{d/2}}\frac{K_{(d/2)-1}\left(\sqrt{2/\lambda }\sqrt{(x-\theta {)}^{\top }{\Sigma }^{-1}(x-\theta )}\right)}{{\left(\sqrt{\lambda /2}\sqrt{(x-\theta {)}^{\top }{\Sigma }^{-1}(x-\theta )}\right)}^{(d/2)-1}},$$

$$f(x|\theta )\asymp \frac{\exp \left(-\sqrt{\frac{2}{\lambda }}\Vert x-\theta {\Vert }_{{\Sigma }^{-1}}\right)}{(\Vert x-\theta {\Vert }_{{\Sigma }^{-1}}{)}^{(d-1)/2}},$$

$$\tau (\alpha ):=\frac{\sqrt{2/(\lambda {\lambda }_{\max })}}{{\left(\sqrt{2/(\lambda {\lambda }_{\min })}+\sqrt{2/(\lambda {\lambda }_{\max })}+C_0\right)}^{1/\alpha }}.$$

$${\left(\log \frac{1}{\overline{h}(p_G,p_{G^{\prime }})}\right)}^{d/(2\alpha )}\exp \left(-\tau (\alpha ){\left(\log \frac{1}{\overline{h}(p_G,p_{G^{\prime }})}\right)}^{1/\alpha }\right)\ge C{W}_2^{2/m}(G,G^{\prime }).$$

$$\displaystyle\Pi\biggr{(}G\in\overline{\mathcal{G}}(\Theta):W_{2}(G,G_{*})\lesssim\exp\left(-\frac{m\tau(\alpha)}{2}\biggr{(}\frac{\log n-\log\log n}{2(d+2)}\biggr{)}^{1/\alpha}\right)\biggr{|}X_{1},\ldots,X_{n}\biggr{)}\to 1$$

$$\displaystyle\sum_{|\kappa|=r}\biggr{|}\biggr{(}\frac{\partial^{|\kappa|}f}{\partial\theta^{\kappa}}(x|\theta_{1})-\frac{\partial^{|\kappa|}f}{\partial\theta^{\kappa}}(x|\theta_{2})\biggr{)}\gamma^{\kappa}\biggr{|}\leq C(x)\|\theta_{1}-\theta_{2}\|^{\delta}\|\gamma\|^{r}$$

Full text

**On posterior contraction of parameters and

interpretability in Bayesian mixture modeling **

[TABLE]

[TABLE]

1 Introduction

Mixture models are one of the most useful tools in a statistician’s toolbox for analyzing heterogeneous data populations. They can be a powerful black-box modeling device to approximate the most complex forms of density functions. Perhaps more importantly, they help the statistician express the data population’s heterogeneous patterns and interpret them in a useful way [33, 30, 34]. The following are common, generic and meaningful questions a practitioner of mixture modeling may ask: (I) how many mixture components are needed to express the underlying latent subpopulations, (II) how efficiently can one estimate the parameters representing these components and, (III) what happens to a mixture model based statistical procedure when the model is actually misspecified?

How to determine the number of mixture components is a question that has long fascinated mixture modelers. Many proposed solutions approached this as a model selection problem. The number of model parameters, hence the number of mixture components, may be selected by optimizing with respect to some regularized loss function; see, e.g., [30, 26, 6] and the references therein. A Bayesian approach to regularization is to place explicitly a prior distribution on the number of mixture components [39, 41, 40, 36]. A convenient aspect of separating out the modeling and inference questions considered in (I) and (II) is that once the number of parameters is determined, the model parameters concerned by question (II) can be estimated and assessed via any standard parametric estimation methods.

In a number of modern applications of mixture modeling to heterogeneous data, such as in topic modeling, the number of mixture components (the topics) may be very large and not necessarily a meaningful quantity [3, 49]. In such situations, it may be appealing for the modeler to consider a nonparametric approach, where both (I) and (II) are considered concurrently. The object of inference is now the mixing measure which encapsulates all unknowns about the mixture density function. There were numerous works exemplifying this approach [29, 13, 24]. In particular, the field of Bayesian nonparametrics (BNP) has offered a wealth of prior distributions on the mixing measure based on which one can arrive at the posterior distribution of any quantity of interest related to the mixing measure [21].

A common choice of such priors is the Dirichlet process [12, 2, 45], resulting in the famous Dirichlet process mixture models [1, 31, 9]. Dirichlet process (DP) and its variants have also been adopted as a building block for more sophisticated hierarchical modeling, thanks to the ease with which computational procedures for posterior inference via Markov Chain Monte Carlo can be implemented [50, 42]. Moreover, there is a well-established asymptotic theory on how such Bayesian nonparametric mixture models result in asymptotically optimal estimation procedures for the population density. See, for instance, [15, 18, 46] for theoretical results specifically on DP mixtures, and [16, 47, 52] for general BNP models. The rich development in both algorithms and theory in the past decades has contributed to the widespread adoption of these models in a vast array of application domains.

For quite some time there was a misconception among quite a few practitioners in various application domains, a misconception that may have initially contributed to their enthusiasm for Bayesian nonparametric modeling, that the use of such nonparametric models eliminates altogether the need for determining the number of mixture components, because the learning of such a quantity is ”automatic” from the posterior samples of the mixing measure. The implicit presumption here is that a consistent estimate of the mixing measure may be equated with a consistent estimate of the number of mixture components. This is not correct, as has been noted, for instance, by [29] in the context of mixing measure estimation. More recently, [35] explicitly demonstrated that the common practice of drawing inference about the number of mixture components via the DP mixture, specifically by reading off the number of support points in the Dirichlet’s posterior sample, leads to an asymptotically inconsistent estimate.

Despite this inconsistency result, it will be shown in this paper that it is still possible to obtain a consistent estimate of the number of mixture components using samples from a Dirichlet process mixtures, or any Bayesian nonparametric mixtures, by applying a simple and fast post-processing procedure on samples drawn from the DP mixture’s posterior. On the other hand, the parametric approach of placing an explicit prior on the number of components yields both a consistent estimate of the number mixture component, and more notably, an optimal posterior contraction rate for component parameters, under a minimal set of conditions. It is worth emphasizing that all these results are possible only under the assumption that the model is well-specified, i.e., the true but unknown population density lies in the support of the induced prior distribution on the mixture densities.

As George Box has said, ”all models are wrong”, but more relevant to us, all mixture models are misspecified in some way. The statistician has a number of modeling decision to make when it comes to mixture models, including the selection of the class of kernel densities, and the support of the space of mixing measures. The significance of question (III) comes to the fore, because if the posterior contraction behavior of model parameters is very slow due to specific modeling choices, one has to be cautious about the interpretability of the parameters of interest. A very slow posterior contraction rate in theory implies that a given data set probably has relatively very slow influence on the movement of mass from the prior to the posterior distribution.

In this paper we study Bayesian estimation of model parameters with both well-specified and misspecified mixture models. There are two sets of results. The first results resolve several outstanding gaps that remain in the existing theory and current practice of Bayesian parameter estimation, given that the mixture model is well-specified. The second set of results describes posterior contraction properties of such procedures when the mixture model is misspecified. We proceed to describe these results, related works and implications to the mixture modeling practice.

1.1 Well-specified regimes

Consider discrete mixing measures $G=\sum_{i=1}^{k}p_{i}\delta_{\theta_{i}}$. Here, $\boldsymbol{p}=(p_{1},\dots,p_{k})$ is a vector of mixing weights, while atoms $\{\theta_{i}\}_{i=1}^{k}$ are elements in a given compact space $\Theta\in\mathbb{R}^{d}$. Mixing measure $G$ is combined with a likelihood function $f(\cdot|\theta)$ with respect to Lebesgue measure $\mu$ to yield a mixture density: $p_{G}(\cdot)=\int f(\cdot|\theta)\mathrm{d}G(\theta)=\sum_{i=1}^{k}p_{i}f(\cdot|\theta_{i})$. When $k<\infty$, we call this a finite mixture model with $k$ components. We write $k=\infty$ to denote an infinite mixture model. The atoms $\theta_{i}$’s are representatives of the underlying subpopulations.

Assume that $X_{1},\ldots,X_{n}$ are i.i.d. samples from a mixture density $p_{G_{0}}(x)=\int f(x|\theta)\textrm{d}G_{0}(\theta)$, where $G_{0}$ is a discrete mixing measure with unknown number of support points $k_{0}<\infty$ residing in $\Theta$. In the overfitted setting, i.e., an upper bound $k_{0}\leq\overline{k}$ is given so that one may work with an overfitted mixture with $\overline{k}$ mixture components, Chen [5] showed that the mixing measure $G_{0}$ can be estimated at a rate $n^{-1/4}$ under the $L_{1}$ metric, provided that the kernel $f$ satisfies a second-order identifiability condition – this is a linear independence property on the collection of kernel function $f$ and its first and second order derivatives with respect to $\theta$.

Asymptotic analysis of Bayesian estimation of the mixing measure that arises in both finite and infinite mixtures, where the convergence is assessed under Wasserstein distance metrics, was first investigated by Nguyen [37]. Convergence rates of the mixing measure under a Wasserstein distance can be directly translated to the convergence rates of the parameters in the mixture model. Under the same (second-order) identifiability condition, it can be shown that either maximum likelihood estimation method or a Bayesian method with a non-informative (e.g., uniform) prior yields a $(\log n/n)^{1/4}$ rate of convergence [22, 37, 24]. Note, however, that $n^{-1/4}$ is not the optimal pointwise rate of convergence. Heinrich and Kahn [20] showed that a distance based estimation method can achieve $n^{-1/2}$ rate of convergence under $W_{1}$ metric, even though their method may not be easy to implement in practice. [23] described a minimum Hellinger distance estimator that achieves the same optimal rate of parameter estimation.

An important question in the Bayesian analysis is whether there exists a suitable prior specification for mixture models according to which the posterior distribution on the mixing measure can be shown to contract toward the true mixing measure at the same fast rate $n^{-1/2}$. Rousseau and Mengersen [43] provided an interesting result in this regard, which states that for overfitted mixtures with a suitable Dirichlet prior on the mixing weights $\boldsymbol{p}$, assuming that an upper bound to the number of mixture component is given, in addition to a second-order type identifiability condition, then the posterior contraction to the true mixing measure can be established by the fact that the mixing weights associated with all redundant atoms of mixing measure $G$ vanish at the rate close to the optimal $n^{-1/2}$.

In our first main result given in Theorem 3.1, we show that an alternative and relatively common choice of prior also yields optimal rates of convergence of the mixing measure (up to a logarithmic term), in addition to correctly recovering the number of mixture components, under considerably weaker conditions. In particular, we study the mixture of finite mixture (MFM) prior, which places an explicit prior distribution on the number of components $k$ and a (conditional) Dirichlet prior on the weights $\boldsymbol{p}$, given each value of $k$. This prior has been investigated by Miller and Harrison [36]. Compared to the method of [43], no upper bound on the true number of mixture components is needed. In addition, only first-order identifiability condition is required for the kernel density $f$, allowing our results to apply to popular mixture models such as location-scale Gaussian mixtures. We also note that the MFM prior is one instance in a class of modeling proposals, e.g., [39, 41, 40] for which the established convergence behavior continues to hold. In other words, from an asymptotic standpoint, all is good on the parametric Bayesian front.

Our second main result, given in Theorem 3.2, is concerned with a Bayesian nonparametric modeling practice. A Bayesian nonparametric prior on mixing measures places zero mass on measures with finite support points, so the BNP model is misspecified with respect to the number of mixture components. Indeed, when $G_{0}$ has only finite support the true density $p_{G_{0}}$ lies at the boundary of the support of the class of densities produced by the BNP prior. Despite the inconsistency results mentioned earlier on the number of mixture components produced by Dirichlet process mixtures, we will show that this situation can be easily corrected by applying a post-processing procedure to the samples generated from the posterior distribution arising from the DP mixtures, or any sufficiently well-behaved Bayesian nonparametric mixture models. By ”well-behaved” we mean any BNP mixtures under which the posterior contraction rate on the mixing measure can be guaranteed by an upper bound using a Wasserstein metric [37].

Our post-processing procedure is simple, and motivated by the observation that a posterior sample of the mixing measure tends to produce a large number of atoms with very small and vanishing weights [19, 35]. Such atoms can be ignored by a suitable truncation procedure. In addition, similar atoms in the metric space $\Theta$ can also be merged in a systematic and probabilistic way. Our procedure, named Merge-Truncate-Merge algorithm, is guaranteed to not only produce a consistent estimate of the number of mixture components but also retain the posterior contraction rates of the original posterior samples for the mixing measure. Theorem 3.2 provides a theoretical basis for the heuristics employed in practice in dealing with mixtures with unknown number of components [19, 40].

1.2 Misspecified regimes

There are several ways a mixture model can be misspecified: either in the kernel density function $f$, or the mixing measure $G$, or both. Thus, in the misspecified setting, we assume that the data samples $X_{1},\ldots,X_{n}$ are i.i.d. samples from a mixture density $p_{G_{0},f_{0}}$, namely, $p_{G_{0},f_{0}}(x)=\int f_{0}(x|\theta)G_{0}(\textrm{d}\theta)$, where both $G_{0}$ and $f_{0}$ are unknown. The statistician draws inference from a mixture model $p_{G,f}$, still denoted by $p_{G}$ for short, where $G$ is a mixing measure with support on compact $\Theta$, and $f$ is a chosen kernel density function. In particular, a Bayesian procedure proceeds by placing a prior on the mixing measure $G$ and obtaining the posterior distribution on $G$ given the $n$-data sample. In general, the true data generating density $p_{G_{0}}$ lies outside the support of the induced prior on $p_{G}$. We study the posterior behavior of $G$ as the sample size $n$ tends to infinity.

The behavior of Bayesian procedures under model misspecification has been investigated in the foundational work of [27, 28]. This body of work focuses primarily on density estimation. In particular, assuming that the true data generating distribution’s density lies outside the support of a Bayesian prior, then the posterior distribution on the model density can be shown to contract to an element of the prior’s support, which is obtained by a Kullback-Leibler (KL) projection of the true density into the prior’s support [27].

It can be established that the posterior of $p_{G}$ contracts to a density $p_{G_{*}}$, where $G_{*}$ is a probability measure on $\Theta$ such that $p_{G_{*}}$ is the (unique) minimizer of the Kullback-Leilber distance $K(p_{G_{0},f_{0}},p_{G})$ among all probability measure $G$ on $\Theta$. This mere fact is readily deduced from the theory of [27], but the outstanding and relevant issue is whether the posterior contraction behavior carries over to that of $G$, and if so, at what rate. In general, $G_{*}$ may not be unique, so posterior contraction of $G$ cannot be established. Under identifiability, $G_{*}$ is unique, but still $G_{*}\neq G_{0}$.

This leads to the question about interpretability when the model is misspecified. Specifically, when $f\neq f_{0}$, it may be unclear how one can interpret the parameters that represent mixing measure $G$, unless $f$ can be assumed to be a reasonable approximation of $f_{0}$. Mixing measure $G$, too, may be misspecified, when the true support of $G_{0}$ may not lie entirely in $\Theta$. In practice, it is a perennial challenge to explicate the relationship between $G_{*}$ and the unknown $G_{0}$. In theory, it is mathematically an interesting question to characterize this relationship, if some assumption can be made on the true $G_{0}$ and $f_{0}$, but this is beyond the scope of this paper. Regardless of the truth about this relationship, it is important for the statistician to know how impactful a particular modeling choice on $f$ and $G$ can affect the posterior contraction rates of the parameters of interest.

The main results that we shall present in Theorem 4.1 and Theorem 4.2 are on the posterior contraction rates of the mixing measure $G$ toward the limit point $G_{*}$, under very mild conditions on the misspecification of $f$. In particular, we shall require that the tail behavior of function $f$ is not much heavier than that of $f_{0}$ (cf. condition (P.5) in Section 4). Specific posterior contraction rates of contraction for $G$ are derived when $f$ is either Gaussian or Laplace density kernel, two representatives for supersmooth and ordinary smooth classes of kernel densities [10]. A key step in our proofs lies in several inequalities which provide upper bound of Wasserstein distances on mixing measures in terms of weighted Hellinger distances, a quantity that plays a fundamental role in the asymptotic characterization of misspecified Bayesian models [27].

It is interesting to highlight that the posterior contraction rate for the misspecified Gaussian location mixture is the same as that of well-specified setting, which is nonetheless extremely slow, in the order of $(1/\log n)^{1/2}$. On the other hand, using a misspecified Laplace location mixture results in some loss in the exponent $\gamma$ of the polynomial rate $n^{-\gamma}$. Although the contrast in contraction rates for the two families of kernels is quite similar to what is obtained for well-specified deconvolution problems for both frequentist methods [10, 55] and Bayesian methods [37, 14], our results are given for misspecified models, which can be seen in a new light: since the model is misspecified anyway, the statistician should be ”free” to choose the kernel that can yield the most favorable posterior contraction for the parameters of his/ her model. In that regard, Laplace location mixtures should always be preferred to a Gaussian location mixtures. Of course it is not always advisable to use a heavy-tailed kernel density function, as dictated by condition (P.5).

The relatively slow posterior contraction rate for $G$ is due to the fact that the limiting measure $G_{*}$ in general may have infinite support, regardless of whether the true $G_{0}$ has finite support or not. From a practical standpoint, it is difficult to interpret the estimate of $G$ if $G_{*}$ has infinite support. However, if $G_{*}$ happens to have a finite number of support points, which is bounded by a known constant, say $\overline{k}$, then by placing a suitable prior on $G$ to reflect this knowledge we show that the posterior of $G$ contracts to $G_{*}$ at a relatively fast rate $(\log n/n)^{1/4}$. This is the same rate obtained under the well-identified setting for overfitted mixtures.

1.3 Further remarks

The posterior contraction theorems in this paper provide an opportunity to re-examine several aspects of the fascinating picture about the tension between a model’s expressiveness and its interpretability. They remind us once again about the tradeoffs a modeler must negotiate for a given inferential goal and the information available at hand. We enumerate a few such insights:

• (1)

”One size does not fit all”: Even though the family of mixture models as a whole can be excellent at inferring about population heterogeneity and at density estimation as a black-box device, a specific mixture model specification cannot do a good job at both. For instance, a Dirichlet process mixture of Gaussian kernels may yield an asymptotically optimal density estimation machine but it performs poorly when it comes to learning of parameters.

• (2)

”Finite versus infinite”: If the number of mixture components is known to be small and an object of interest, then employing an explicit prior on this quantity results in the optimal posterior contraction rate for the model parameters and thus is a preferred method. When this quantity is known to be high or not a meaningful object of inference, Bayesian nonparametric mixtures provide a more attractive alternative as it can flexibly adapt to complex forms of densities. Regardless, one can still consistently recover the true number of mixture components using a nonparametric approach.

• (3)

”Some forms of misspecification are more useful than others”. When the mixture model is misspecified, careful design choices regarding the (mispecified) kernel density and the support of the mixing measure can significantly speed up the posterior contraction behavior of model parameters. For instance, a heavy-tailed and ordinary smooth kernel such as the Laplace, instead of the Gaussian kernel, is shown to be especially amenable to efficient parameter estimation.

The remainder of the paper is organized as follows. Section 2 provides necessary backgrounds about mixture models, Wasserstein distances and several key notions of strong identifiability. Section 3 presents posterior contraction theorems for well-mispecified mixture models for both parametric and nonparametric Bayesian models. Section 4 presents posterior contraction theorems when the mixture model is misspecified. In Section 5, we provide illustrations of the Merge-Truncate-Merge algorithm via a simulation study. Proofs of key results are provided in Section 6 while the remaining proofs are deferred to the Appendices.

Notation

Given two densities $p,q$ (with respect to the Lebesgue measure $\mu$), the total variation distance is given by $V(p,q)={\displaystyle(1/2)\int{|p(x)-q(x)|}\textrm{d}\mu(x)}$. Additionally, the squared Hellinger distance is given by $h^{2}(p,q)=\\ {\displaystyle(1/2)\int(\sqrt{p(x)}-\sqrt{q(x)})^{2}\textrm{d}\mu(x)}$. Furthermore, the Kullback-Leibler (KL) divergence is given by $K(p,q)={\displaystyle\int\log(p(x)/q(x))p(x)\textrm{d}\mu(x)}$ and the squared KL divergence is given by $K_{2}(p,q)={\displaystyle\int\log(p(x)/q(x))^{2}p(x)\textrm{d}\mu(x)}$. For a measurable function $f$, let $Qf$ denote the integral $\int fdQ$. For any $\kappa=(\kappa_{1},\ldots,\kappa_{d})\in\mathbb{N}^{d}$, we denote $\dfrac{\partial^{|\kappa|}{f}}{\partial{\theta^{\kappa}}}(x|\theta)=\dfrac{\partial^{|\kappa|}{f}}{\partial{\theta_{1}^{\kappa_{1}}\ldots\partial{\theta_{d}^{\kappa_{d}}}}}(x|\theta)$ where $\theta=(\theta_{1},\ldots,\theta_{d})$. For any metric $d$ on $\Theta$, we define the open ball of $d$-radius $\epsilon$ around $\theta_{0}\in\Theta$ as $B_{d}(\epsilon,\theta_{0})$. We use $D(\epsilon,\Omega,\tilde{d})$ to denote the maximal $\epsilon$-packing number for a general set $\Omega$ under a general metric $\tilde{d}$ on $\Omega$. Additionally, the expression $a_{n}\gtrsim b_{n}$ will be used to denote the inequality up to a constant multiple where the value of the constant is independent of $n$. We also denote $a_{n}\asymp b_{n}$ if both $a_{n}\gtrsim b_{n}$ and $a_{n}\lesssim b_{n}$ hold. Furthermore, we denote $A^{c}$ as the complement of set $A$ for any set $A$ while $B(x,r)$ denotes the ball, with respect to the $l_{2}$ norm, of radius $r>0$ centered at $x\in\mathbb{R}^{d}$. Finally, we use $\text{Diam}(\Theta)=\sup\{\|\theta_{1}-\theta_{2}\|:\theta_{1},\theta_{2}\in\Theta\}$ to denote the diameter of a given parameter space $\Theta$ relative to the $l_{2}$ norm, $\|\cdot\|$, for elements in $\mathbb{R}^{d}$.

2 Preliminaries

We recall the notion of Wasserstein distance for mixing measures, along with the notions of strong identifiability and uniform Lipschitz continuity conditions that prove useful in Section 3.

Mixture model

Throughout the paper, we assume that $X_{1},\ldots,X_{n}$ are i.i.d. samples from a true but unknown distribution $P_{G_{0}}$ with given density function

[TABLE]

where $G_{0}=\sum\limits_{i=1}^{k_{0}}{p_{i}^{0}\delta_{\theta_{i}^{0}}}$ is a true but unknown mixing distribution with exactly $k_{0}$ number of support points, for some unknown $k_{0}$. Also, $\left\{f(x|\theta),\theta\in\Theta\subset\mathbb{R}^{d}\right\}$ is a given family of probability densities (or equivalently kernels) with respect to a sigma-finite measure $\mu$ on $\mathcal{X}$ where $d\geq 1$. Furthermore, $\Theta$ is a chosen parameter space, where we empirically believe that the true parameters belong to. In a well-specified setting, all support points of $G_{0}$ reside in $\Theta$, but this may not be the case in a misspecified setting.

Regarding the space of mixing measures, let $\mathcal{E}_{k}:=\mathcal{E}_{k}(\Theta)$ and $\mathcal{O}_{k}:=\mathcal{O}_{k}(\Theta)$ respectively denote the space of all mixing measures with exactly and at most $k$ support points, all in $\Theta$. Additionally, denote $\mathcal{G}:=\mathcal{G}(\Theta)=\mathop{\cup}\limits_{k\in\mathbb{N}_{+}}{\mathcal{E}_{k}}$ the set of all discrete measures with finite supports on $\Theta$. Moreover, $\overline{\mathcal{G}}(\Theta)$ denotes the space of all discrete measures (including those with countably infinite supports) on $\Theta$. Finally, $\mathcal{P}(\Theta)$ stands for the space of all probability measures on $\Theta$.

Wasserstein distance

As in [37, 22] it is useful to analyze the identifiability and convergence of parameter estimation in mixture models using the notion of Wasserstein distance, which can be defined as the optimal cost of moving masses transforming one probability measure to another [51]. Given two discrete measures $G=\mathop{\sum}\limits_{i=1}^{k}{p_{i}\delta_{\theta_{i}}}$ and $G^{\prime}=\sum_{i=1}^{k^{\prime}}p^{\prime}_{i}\delta_{\theta^{\prime}_{i}}$, a coupling between $\boldsymbol{p}$ and $\boldsymbol{p^{\prime}}$ is a joint distribution $\boldsymbol{q}$ on $[1\ldots,k]\times[1,\ldots,k^{\prime}]$, which is expressed as a matrix $\boldsymbol{q}=(q_{ij})_{1\leq i\leq k,1\ \leq j\leq k^{\prime}}\in[0,1]^{k\times k^{\prime}}$ with marginal probabilities $\mathop{\sum}\limits_{i=1}^{k}{q_{ij}}=p_{j}^{\prime}$ and $\mathop{\sum}\limits_{j=1}^{k^{\prime}}{q_{ij}}=p_{i}$ for any $i=1,2,\ldots,k$ and $j=1,2,\ldots,k^{\prime}$. We use $\mathcal{Q}(\boldsymbol{p},\boldsymbol{p^{\prime}})$ to denote the space of all such couplings. For any $r\geq 1$, the $r$-th order Wasserstein distance between $G$ and $G^{\prime}$ is given by

[TABLE]

where $\|\cdot\|$ denotes the $l_{2}$ norm for elements in $\mathbb{R}^{d}$. It is simple to see that if a sequence of probability measures $G_{n}\in\mathcal{O}_{k_{0}}$ converges to $G_{0}\in\mathcal{E}_{k_{0}}$ under the $W_{r}$ metric at a rate $\omega_{n}=o(1)$ for some $r\geq 1$ then there exists a subsequence of $G_{n}$ such that the set of atoms of $G_{n}$ converges to the $k_{0}$ atoms of $G_{0}$, up to a permutation of the atoms, at the same rate $\omega_{n}$.

Strong identifiability and uniform Lipschitz continuity

The key assumptions that will be used to analyze the posterior contraction of mixing measures include uniform Lipschitz condition and strong identifiability condition. The uniform Lipschitz condition can be formulated as follows [22].

Definition 2.1**.**

We say the family of densities $\left\{f(x|\theta),\theta\in\Theta\right\}$ is uniformly Lipschitz up to the order $r$, for some $r\geq 1$, if $f$ as a function of $\theta$ is differentiable up to the order $r$ and its partial derivatives with respect to $\theta$ satisfy the following inequality

[TABLE]

for any $\gamma\in\mathbb{R}^{d}$ and for some positive constants $\delta$ and $C$ independent of $x$ and $\theta_{1},\theta_{2}\in\Theta$. Here, $\gamma^{\kappa}=\prod\limits_{i=1}^{d}{\gamma_{i}^{\kappa_{i}}}$ where $\kappa=(\kappa_{1},\ldots,\kappa_{d})$.

The first order uniform Lipschitz condition is satisfied by many popular classes of density functions, including Gaussian, Student’s t, and skew-normal family. Now, strong identifiability condition of the $r^{th}$ order is formulated as follows,

Definition 2.2**.**

For any $r\geq 1$, we say that the family $\left\{f(x|\theta),\theta\in\Theta\right\}$ (or in short, $f$) is identifiable in the order $r$, for some $r\geq 1$, if $f(x|\theta)$ is differentiable up to the order $r$ in $\theta$ and the following holds

• A1.

For any $k\geq 1$, given $k$ different elements $\theta_{1},\ldots,\theta_{k}\in\Theta$. If we have $\alpha_{\eta}^{(i)}$ such that for almost all $x$

[TABLE]

then $\alpha_{\eta}^{(i)}=0$ for all $1\leq i\leq k$ and $|\eta|\leq r$.

Many commonly used families of density functions satisfy the first order identifiability condition, including location-scale Gaussian distributions and location-scale Student’s t-distributions. Technically speaking, strong identifiability conditions are useful in providing the guarantee that we have some sort of lower bounds of Hellinger distance between mixing densities in terms of Wasserstein metric between mixing measures. For example, if $f$ is identifiable in the first order, we have the following inequality [22]

[TABLE]

for any $G\in\mathcal{E}_{k_{0}}$. It implies that for any estimation method that yields the convergence rate $n^{-1/2}$ for density $p_{G_{0}}$ under the Hellinger distance, the induced rate of convergence for the mixing measure $G_{0}$ is $n^{-1/2}$ under $W_{1}$ distance.

3 Posterior contraction under well-specified regimes

In this section, we assume that the mixture model is well-specified, i.e., the data are i.i.d. samples from the mixture density $p_{G_{0}}$, where mixing measure $G_{0}$ has $k_{0}$ support points in compact parameter space $\Theta\subset\mathbb{R}^{d}$. Within this section, we assume further that the true but unknown number of components $k_{0}$ is finite. A Bayesian modeler places a prior distribution $\Pi$ on a suitable subspace of $\overline{\mathcal{G}}(\Theta)$. Then, the posterior distribution over $G$ is given by:

[TABLE]

We are interested in the posterior contraction behavior of $G$ toward $G_{0}$, in addition to recovering the true number of mixture components $k_{0}$.

3.1 Prior results

The customary prior specification for a finite mixture is to use a Dirichlet distribution on the mixing weights and another standard prior distribution on the atoms of the mixing measure. Let $H$ be a distribution with full support on $\Theta$. Thus, for a mixture of $k$ components, the full Bayesian mixture model specification takes the form:

[TABLE]

Suppose for a moment that $k_{0}$ is known, we can set $k=k_{0}$ in the above model specification. Thus we would be in an exact-fitted setting. Provided that $f$ satisfies both first-order identifiability condition and the uniform Lipschitz continuity condition, $H$ is approximately uniform on $\Theta$, then according to [37, 22] it can be established that as $n$ tends to infinity,

[TABLE]

The $(\log n/n)^{1/2}$ rate of posterior contraction is optimal up to a logarithmic term.

When $k_{0}$ is unknown, there may be a number of ways for the modeler to proceed. Suppose that an upper bound of $k_{0}$ is given, say $k_{0}<\overline{k}$. Then by setting $k=\overline{k}$ in the above model specification, we have a Bayesian overfitted mixture model. Provided that $f$ satisfies the second-order identifability condition and the uniform Lipschitz continuity condition, $H$ is again approximately uniform distribution on $\Theta$, then it can be established that [37, 22]:

[TABLE]

This result does not provide any guarantee about whether the true number of mixture components $k_{0}$ can be recovered. The rate (upper bound) $(\log n/n)^{1/4}$ under $W_{2}$ metric implies that under the posterior distribution the redundant mixing weights of $G$ contracts toward zero at the rate $(\log n/n)^{1/2}$, but the posterior contraction to each of the $k_{0}$ atoms of $G_{0}$ occurs at the rate $(\log n/n)^{1/4}$ only.

Interestingly, it can be shown by Rousseau and Mengersen [43] that with a more judicious choice of prior distribution on the mixing weights, one can achieve a near-optimal posterior contraction behavior. Specifically, they continued to employ the Dirichlet prior, but they required the Dirichlet’s hyperparameters set to be sufficiently small: $\gamma/k\leq d/2$ in (3.1) where $k=\overline{k}$, $d$ is the dimension of the parameter space $\Theta$. Then, under some conditions on kernel $f$ approximately comparable to the second-order identifiability and the uniform Lipschitz continuity condition defined in the previous section, they showed that for any $\epsilon>0$, as $n$ tends to infinity

[TABLE]

For a more precise statement along with the complete list of sufficient conditions leading to claim (6), we refer the reader to the original theorem of [43]. Although their theorem is concerned with only the behavior of the redundant mixing weights $p_{i}$, where $i\in I$, which vanish at a near-optimal rate $n^{-1/2+\epsilon}$, it can be deduced from their proof that the posterior contraction for the true atoms of $G_{0}$ occurs at this near-optimal rate as well. [43] also showed that this performance may not hold if the Dirichlet’s hyperparameters are set to be sufficiently large. Along this line, concerning the recovery of the number of mixture components $k_{0}$, [4] demonstrated the convergence of the posterior mode of the number of components to the true number of components $k_{0}$ at a rate $n^{-\rho}$, where $\rho$ depends on $\overline{k}-k_{0}$, the number of redundant components forced upon by our model specification.

3.2 Optimal posterior contraction via a parametric Bayesian mixture

We will show that optimal posterior contraction rates for mixture model parameters can be achieved by a natural Bayesian extension on the prior specification, even when the upper bound on the number of mixture component $k$ is unknown. The modeling idea is simple and truly Bayesian in spirit: since $k_{0}$ is unknown, let $K$ be a natural-valued random variable representing the number of mixture components. We endow $K$ with a suitable prior distribution $q_{K}$ on the positive integers. Conditioning on $K=k$, for each $k$, the model is specified as before:

[TABLE]

This prior specification is called mixture of finite mixtures (MFM) model [41, 48, 36]. In the sequel we show that the application of the MFM prior leads to the optimal posterior contraction rates for the model parameters. Interestingly, such guarantees can be established under very mild conditions on the kernel density $f$: only the uniform Lipschitz continuity and the first-order identifiability conditions will be required. The first-order identifiability condition is the minimal condition for which the optimal posterior contraction rate can be established, since this condition is also necessary for exact-fitted mixture models to receive the $n^{-1/2}$ posterior contraction rate. We proceed to state such conditions.

• (P.1)

The parameter space $\Theta$ is compact, while kernel density $f$ is first-order identifiable and admits the uniform Lipschitz property up to the first order.

• (P.2)

The base distribution $H$ is absolutely continuous with respect to the Lebesgue measure $\mu$ on $\mathbb{R}^{d}$ and admits a density function $g(\cdot)$. Additionally, $H$ is approximately uniform, i.e., $\min_{\theta\in\Theta}g(\theta)>c_{0}>0$.

• (P.3)

There exists $\epsilon_{0}>0$ such that ${\displaystyle\int(p_{G_{0}}(x))^{2}/p_{G}(x)d\mu(x)\leq M(\epsilon_{0}})$ as long as $W_{1}(G,G_{0})\leq\epsilon_{0}$ for any $G\in\mathcal{O}_{k_{0}}$ where $M(\epsilon_{0})$ depends only on $\epsilon_{0}$, $G_{0}$, and $\Theta$.

• (P.4)

The prior $q_{K}$ places positive mass on the set of natural numbers, i.e., $q_{K}(k)>0$ for all $k\in\mathbb{N}$.

Theorem 3.1**.**

Under assumptions (P.1), (P.2), (P.3), and (P.4) on MFM, we have that

• (a)

$\Pi(K=k_{0}|X_{1},\ldots,X_{n})\to 1$* a.s. under $P_{G_{0}}$.*

• (b)

Moreover,

[TABLE]

in $P_{G_{0}}$-probability.

The proof of Theorem 3.1 is deferred to Section 6.1. We make several remarks regarding the conditions required in the theorem. It is worth stating up front that these conditions are almost minimal in order for the optimal posterior contraction to be guaranteed, and are substantially weaker than previous works (as discussed above). Assumption (P.1) is crucial in establishing that the Hellinger distance $h(p_{G},p_{G_{0}})\geq C_{0}W_{1}(G,G_{0})$ where $C_{0}$ is some positive constant depending only on $G_{0}$ and $\Theta$. Assumption (P.2) and (P.4) are standard conditions on the support of the prior so that posterior consistency can be guaranteed for any unknown $G_{0}$ with unknown number of support atoms residing on $\Theta$. Finally, the role of (P.3) is to help control the growing rate of KL neighborhood, which is central in the analysis of posterior convergence rate of mixing measures. This assumption is held for various choices of kernel $f$, including location families and location-scale families. Therefore, the assumptions (P.1), (P.2),(P.3) and (P.4) are fairly general and satisfied by most common choice of kernel densities.

Further remarks

Theorem 3.1 provides a positive endorsement for employing the MFM prior when the number of mixture components is unknown, but is otherwise believed to be finite and an important quantity of inferential interest. The papers of [41, 36] discuss additional favorable properties of this class of models. However, when the true number of mixture components is large, posterior inference with the MFM may still be inefficient in practice. This is because much of the computational effort needs to be expended for the model selection phase, so that the number of mixture components can be reliably ascertained. Only then does the fast asymptotic rate of parameter estimation come meaningfully into effect.

3.3 A posteriori processing for BNP mixtures

Instead of placing a prior distribution explicitly on the number of mixture components when this quantity is unknown, another predominant approach is to place a Bayesian nonparametric prior on the mixing measure $G$, resulting in infinite mixture models. Bayesian nonparametric models such as Dirichlet process mixtures and the variants have remarkably extended the reach of mixture modeling into a vast array of applications, especially those areas where the number of mixture components in the modeling is very large and difficult to fathom, or when it is a quantity of only tangential interest. For instance, in topic modeling applications of web-based text corpora, one may be interested in the most ”popular” topics, the number of topics is totally meaningless [3, 50, 38, 54]. DP mixtures and variants can also serve as an asymptotically optimal device for estimating the population density, under standard conditions on the true density’s smoothness, see, e.g., [17, 18, 46, 44].

Since a nonparametric Bayesian prior such as the Dirichlet process places zero probability on mixing measures with finite number of supporting atoms, the Dirichlet process mixture’s posterior is inconsistent on the number of mixture components, provided the true number of mixture components is finite [35]. It is well known in practice that Dirichlet process mixtures tend to produce many small extraneous components around the ”true” clusters, making them challenging to use to draw conclusion about the true number of mixture components when this becomes a quantity of interest [32, 19]. In this section we describe a simple posteriori processing algorithm that consistently estimates the number of components for any general Bayesian prior, even without the exact knowledge of its structure as long as the posterior for that prior contracts at some known rate to the true $G_{0}$.

Our starting point is the availability of a mixing measure sample $G$ that is drawn from the posterior distribution $\Pi(G|X_{1},\ldots,X_{n})$, where $X_{1},\ldots,X_{n}$ are i.i.d. samples of the mixing density $p_{G_{0}}$. Under certain conditions on the kernel density $f$, it can be established that for some Wasserstein metric $W_{r}$, as $n\rightarrow\infty$

[TABLE]

for all constant $\delta>0$, while $\omega_{n}=o(1)$ is a vanishing rate. Thus, $\omega_{n}$ can be taken to be (slightly) slower than actual rate of posterior contraction of the mixing measure. Concrete examples of the posterior contraction rates in infinite and (overfitted) finite mixtures are given in [37, 14, 22].

The posterior processing algorithm operates on an instance of mixing measure $G$, by suitably merging and truncating atoms that provide the support for $G$. The only inputs to the algorithm, which we call Merge-Truncate-Merge (MTM) algorithm is $G$, in addition to the upper bound of posterior contraction rate $\omega_{n}$, and a tuning parameter $c>0$. The tuning parameter $c$ is useful in practice, as we shall explain, but in theory the algorithm ”works” for any constant $c>0$. Thus, the method is almost ”automatic” as it does not require any additional knowledge about the kernel density $f$ or the space of support $\Theta$ for the atoms. It is also simple and fast. We shall show that the outcome of the algorithm is a consistent estimate of both the number of mixing components and the mixing measure. The latter admits a posterior contraction rate’s upper bound $\omega_{n}$ as well.

The detailed pseudocode of MTM algorithm is summarized in Algorithm 1. At a high level, it consists of two main stages. The first stage involves a probabilistic procedure for merging atoms that may be clustered near one another. The second stage involves a deterministic procedure for truncating extraneous atoms and merging them suitably with the remaining ones in a systematic way. The driving force of the algorithm lies in the asymptotic bound on the Wasserstein distance, i.e., $W_{r}(G,G_{0})\leq c\omega_{n}$ with high probability. When $c\omega_{n}$ is sufficiently small, there may be many atoms that concentrate around each of the supporting atoms of $G_{0}$. Although $G_{0}$ is not known, such clustering atoms may be merged into one, by our first stage of probabilistic merging scheme. The second stage (truncate-merge) is also necessary in order to obtain a consistent estimate of $k_{0}$, because there remain distant atoms which carry a relatively small amount of mass. They will need to be suitably truncated and merged with the other more heavily supported atoms. In other words, our method can be viewed as a formal procedure of the common practices employed by numerous practitioners.

We proceed to present the theoretical guarantee for the outcome of Algorithm 1.

Theorem 3.2**.**

Let $G$ be a posterior sample from posterior distribution of any Bayesian procedure, namely, $\Pi(\cdot|X_{1},\ldots,X_{n})$ according to which the upper bound (9) holds for all $\delta>0$. Let $\widetilde{G}$ and $\tilde{k}$ be the outcome of Algorithm 1 applied to $G$, for an arbitrary constant $c>0$. Then the following hold as $n\rightarrow\infty$.

1. (a)

$\Pi(\tilde{k}=k_{0}|X_{1},\ldots,X_{n})\to 1$* in $P_{G_{0}}$-probability.* 2. (b)

For all $\delta>0$, \Pi\biggr{(}G\in\overline{\mathcal{G}}(\Theta):W_{r}(\tilde{G},G_{0})\leq\delta\omega_{n}\biggr{|}X_{1},\ldots,X_{n}\biggr{)}\longrightarrow 1 in $P_{G_{0}}$-probability.

We add several comments concerning this theorem.

• (i)

The proof of this theorem is deferred to Section 6.2, where we clarify carefully the roles played by each step of the MTM algorithm.

• (ii)

Although it is beyond the scope of this paper to study the practical viability of the MTM algorithm, for interested readers we present a brief illustration of the algorithm via simulations in Section 5.

• (iii)

In practice, one may not have a mixing measure $G$ sampled from the posterior $\Pi(\cdot|X_{1},\ldots,X_{n})$ but a sample from $G$ itself, say $F_{n}$. Then one can apply the MTM algorithm to $F_{n}$ instead. Assume that $F_{n}$ is sufficiently close to $G$, in the sense that $W_{r}(F_{n},G)\lesssim W_{r}(G,G_{0})$, it is straightforward to extend the above theorem to cover this scenario.

Further remarks

At this point, one may look forward to some guidance regarding the modeling choices of parametrics versus nonparametrics. Even in the tight arena of Bayesian mixture modeling, the jury may still be out. The results in this section seems to provide a stronger theoretical support for the former, when it comes to the efficiency of parameter estimation and the corresponding model interpretation.

However, as we will see in the next section, when the mixture model is misspecified, the fast posterior contraction rate offered by the use of the MFM prior is no longer valid. On the other hand, Bayesian nonparametric models are more versatile in adapting to complex forms of population densities. In many modern applications it is not meaningful to estimate the number of mixing components, only the most ”significant” ones in a sense suitably defined. Perhaps a more meaningful question concerning a Bayesian nonparametric mixture model is whether it is capable of learning selected mixture components in an efficient way.

4 Posterior contraction under model misspecification

In this section, we study the posterior contraction behavior of the mixing measure under the realistic scenarios of model misspecification. There are several ways a mixture model can be misspecified, due to the misspecification of the kernel density function $f$, or the support of the mixing measure $G$, or both. From here on, we shall assume that the data population follows a mixture distribution composed of unknown kernel density $f_{0}$ and unknown mixing measure $G_{0}$ — thus, in this section the true density shall be denoted by $p_{G_{0},f_{0}}$ to highlight the possibility of misspecification.

To avoid heavy subscripting, we continue to use $p_{G}$ instead of $p_{G,f}$ to represent the density function of the mixture model that we operate on. The kernel density $f$ is selected by the modeler. Additionally, $G$ is endowed with a suitable prior $\Pi$ on the space of mixing measures with support belonging to compact parameter space $\Theta$. By Bayes rule (Eq. (2)) one obtains the posterior distribution $\Pi(G|X_{1},\ldots,X_{n})$, where the $n$-i.i.d. sample $X_{1},\ldots,X_{n}$ are generated by $p_{G_{0},f_{0}}$. It is possible that $f\neq f_{0}$. It is also possible that the support of $G_{0}$ does not reside within $\Theta$. In practice, the statistical modeler would hope that the kernel choice of $f$ is not too different from the true but unknown $f_{0}$. Otherwise, it would be unclear how one can interpret the parameters that represent the mixing measure $G$. Our goal is to investigate the posterior contraction of $\Pi(G|X_{1},\ldots,X_{n})$ in such situations, as sample size $n$ tends to infinity. The theory is applicable for a broad class of prior specification on the mixing measures on $\Theta$, including the MFM prior and a nonparametric Bayesian prior such as the Dirichlet process.

A fundamental quantity that arises in the theory of Bayesian misspecification for density estimation is the minimizer of the Kullback-Leibler (KL) distance from the true population density to a density function residing in the support of the induced prior on the space of densities $p_{G}$, which we shall assume to exist (cf. [27]). Moreover, assume that the KL minimizer can be expressed as a mixture density $p_{G_{*}}$, where $G_{*}$ is a probability measure on $\Theta$. We may write

[TABLE]

We will see in the sequel that the existence of the KL minimizer $p_{G^{*}}$ entails its uniqueness. In general, however, $G_{*}$ may be non-unique. Thus, define

[TABLE]

It is challenging to characterize the set $\mathcal{M}^{*}$ in general. However, a very useful technical property can be shown as follows:

Lemma 4.1**.**

For any $G\in\mathcal{P}(\Theta)$ and $G_{*}\in\mathcal{M}^{*}$, it holds that ${\displaystyle\int\dfrac{p_{G}(x)}{p_{G_{*}}(x)}p_{G_{0},f_{0}}(x)\mathrm{d}x\leq 1}$.

By exploiting the fact that the class of mixture densities is a convex set, the proof of this lemma is similar to that of Lemma 2.3 of [27], so it is omitted. This leads quickly to the following fact.

Lemma 4.2**.**

For any two elements $G_{1,*},G_{2,*}\in\mathcal{M}^{*}$, $p_{G_{1,*}}(x)=p_{G_{2,*}}(x)$ for almost all $x\in\mathcal{X}$.

In other words, the mixture density $p_{G_{*}}$ is uniquely identifiable. Under a standard identifiability condition of the kernel $f$, which is satisfied by the examples considered in this section, it follows that $G_{*}$ is unique. Due to the model misspecification, in general $G_{*}\neq G_{0}$. The best we can hope for is that the posterior distribution of the mixing measure $G$ contracts toward $G_{*}$ as $n$ tends to infinity.

The goal of the remaining of this section is to study the posterior contraction behavior of the (misspecified) mixing measure $G$ towards the unique $G_{*}$.

Following the theoretical framework of [27] and [37], the posterior contraction behavior of the mixing measure $G$ can be obtained by studying the relationship of a weighted version of Hellinger distance and corresponding Wasserstein distances between $G$ and the limiting point $G_{*}$. In particular, for a fixed pair of mixture densities $p_{G_{0},f_{0}}$ and $p_{G_{*}}$, the weighted Hellinger $\overline{h}$ between two mixture densities is defined as follows [27].

Definition 4.1**.**

For $G_{1},G_{2}\in\mathcal{P}(\Theta)$,

[TABLE]

It is clear that when $G_{*}=G_{0}$ and $f=f_{0}$, the weighted Hellinger distance reduces to the standard Hellinger distance. In general they are different due to misspecification. According to Lemma 4.1, we have $\overline{h}(p_{G_{1}},p_{G_{2}})\leq 1$ for all $G_{1},G_{2}\in\mathcal{P}(\Theta)$.

Choices of prior on mixing measures

As in the previous section, we work with two representative priors on the mixing measure: the MFM prior and the Dirichlet process prior. Both prior choices may contribute to the model misspecification, if the true mixing measure $G_{0}$ lies outside of the support of the prior distribution.

Recall the MFM prior specification given in Eq. (7). We also need a stronger condition on $q_{K}$:

• (P.4’)

The prior distribution $q_{K}$ on the number of components satisfies $q_{k}\gtrsim k^{-\alpha_{0}}$ for some $\alpha_{0}>1$.

Note that the assumption with prior on the number of components $q_{K}$ is mild and satisfied by many distributions, such as Poisson distribution. In order to obtain posterior contraction rates, one needs to make sure the prior places sufficient mass on the (unknown) limiting point of interest. For the MFM prior, such a condition is guaranteed by the following lemma.

Lemma 4.3**.**

Let $\Pi$ denote the prior for generating $G$ based on MFM (7), where $H$ admits condition (P.2) and $q_{K}$ admits (P.4’). Fix $r\geq 1$. Then the following holds, for any $G_{*}\in\mathcal{P}(\Theta)$

[TABLE]

for all $\epsilon$ sufficiently small so that $D(\epsilon,\Theta,\|.\|)>\gamma$. Here, $D=D(\epsilon,\Theta,\|.\|)$ and $q_{D}$ stand for the maximal $\epsilon$-packing number for $\Theta$ under $\|.\|$ norm and the prior weight $\Pi(K=D)$, respectively.

The proof of Lemma 4.3 is provided in Section 6.3. Alternatively, for a Dirichlet process prior, $G$ is distributed a priori according to a Dirichlet measure with concentration parameter $\gamma>0$ and base measure $H$ satisfying condition (P.2). An analogous concentration bound for such a prior is given in Lemma 5 of [37].

It is somewhat interesting to note that the difference in the choices of prior under misspecification does not affect the posterior contraction bounds that we can establish. In particular, as we have seen for the definition, $G_{*}$ does not depend on a specific choice of prior distribution (only its support). Due to misspecification, $G_{*}$ may have infinite support, even if the true $G_{0}$ has a finite number of support points. When $G_{*}$ has infinite support, the posterior contraction toward $G_{*}$ becomes considerably slower compared to the well-specified setting. In addition to the structure of $G_{*}$, we will see in the sequel that the modeler’s specific choice of kernel density $f$ proves to be especially impactful on the rate of posterior contraction.

4.1 Gaussian location mixtures

Consider a class of kernel densities that belong to the supersmooth location family of density functions. A particular example that we focus on in this section is a class of Gaussian distributions with some fixed covariance matrix $\Sigma$. More precisely, $f$ has the following form:

[TABLE]

where $|\cdot|$ stands for matrix determinant. Note that, Gaussian kernel is perhaps the most popular choice in mixture modeling.

With the Gaussian location kernel, it is possible to obtain a lower bound on the Hellinger distance between the mixture densities in terms of the Wasserstein distance between corresponding mixing measures [37]. More useful in the misspecified setting is a key lower bound for the weighted Hellinger distance in terms of the Wasserstein metric, which is given as follows. We shall require a technical condition relating $f$ to the true $f_{0}$ and $G_{0}$:

• (P.5)

The support of $G_{0}$, namely, $\textrm{supp}(G_{0})$ is a bounded subset of $\mathbb{R}^{d}$. Moreover, there are some constants $C_{0},C_{1},\alpha>0$ such that for any $R>0$,

[TABLE]

Proposition 4.1**.**

Let $f$ be a Gaussian kernel given by (12), $\Theta$ a bounded subset of $\mathbb{R}^{d}$. Moreover, assume that $f$ satisfies condition (P.5) for $\alpha\leq 2$. Then, there exists $\epsilon_{0}>0$ depending on $\Theta$ and $\Sigma$, such that for any $G,G^{\prime}\in\mathcal{P}(\Theta)$, whenever $\overline{h}(p_{G},p_{G^{\prime}})\leq\epsilon_{0}$, the following inequality holds

[TABLE]

Here, $\lambda_{\max}$ and $\lambda_{\min}$ are respectively the maximum and minimum eigenvalue of $\Sigma$. $C$ is a constant depending on the parameter space $\Theta$, the dimension $d$, the covariance matrix $\Sigma$, $G_{0}$ and $C_{1}$ in condition (P.5).

The proof of Proposition 4.1 is provided in Section 6.4.

We are ready to prove the first main result of this section.

Theorem 4.1**.**

Assume that $f$ satisfies condition specified in Prop. 4.1, and $\Pi$ is an MFM prior on $\mathcal{P}(\Theta)$ specified in Lemma 4.3. Then, as $n$ tends to infinity,

[TABLE]

in $p_{G_{0},f_{0}}$-probability.

The proof of Theorem 4.1 is given in Section 9.1. The same posterior contraction behavior holds if we replace MFM prior by the Dirichlet Process prior with no change in the proof, except that Lemma 5 of [37] is used in place of Lemma 4.3.

4.2 Laplace location mixtures

Next, we consider a class of multivariate Laplace kernel, a representative in the family of ordinary smooth density functions. It was shown by [37] that under a Dirichlet process location mixture with a Laplace kernel, assume the model is well-specified, the posterior contraction rate of mixing measures to $G_{0}$ is of order $n^{-\gamma}$ for some constant $\gamma>0$. Under the current misspecification setting, we will be able to derive contraction rates toward $G_{*}$ in the order of $n^{-\gamma^{\prime}}$ for some constant $\gamma^{\prime}$ dependent on $\gamma$. The density of location Laplace distributions is given by :

[TABLE]

where $\Sigma$ and $\lambda>0$ are respectively fixed covariance matrix and scale parameter such that $|\Sigma|=1$. Here, $K_{v}$ is a Bessel function of the second kind of order $v$. As discussed in [8], $K_{m}(x)\sim\sqrt{\frac{\pi}{2x}}\exp(-x)$ as $|x|\rightarrow\infty$. Therefore, there exists $\tilde{R}$ such that as long as $\|x-\mu\|>\tilde{R}$, we have

[TABLE]

where we use the shorthand notation $\|y\|_{\Sigma^{-1}}=\sqrt{y^{\top}\Sigma^{-1}y}$. To ease the ensuing presentation, we denote

[TABLE]

The following proposition provides a key lower bound of weighted Hellinger distance in terms of the Wasserstein metric.

Proposition 4.2**.**

Let $f$ be a Laplace kernel given by (13) for fixed $\Sigma$ and $\lambda$ such that $|\Sigma|=1$. Moreover, $f$ satisfies condition (P.5) for some $\alpha\geq 1$. Then, there exists $\epsilon_{0}>0$ depending on $\Theta$, $\lambda$ and $\Sigma$, such that for any $G,G^{\prime}\in\mathcal{P}(\Theta)$, whenever $\overline{h}(p_{G},p_{G^{\prime}})\leq\epsilon_{0}$, the following inequality holds

[TABLE]

for any positive constant $m<4/(4+5d)$. Here, $\lambda_{\max}$ and $\lambda_{\min}$ are respectively the maximum and minimum eigenvalue of $\Sigma$. The constant $C$ depends on the parameter space $\Theta$, the dimension $d$, the covariance matrix $\Sigma$, the scale parameter $\lambda$, $G_{0}$ and $C_{1}$ in (P.5).

The proof of Proposition 4.2 is provided in Section 6.5. Given the above result, the posterior contraction rate for mixing measures $G$ in the location family of Laplace mixture distributions can be obtained from the following result:

Theorem 4.2**.**

Assume that $f$ is given by equation (13) for fixed $\Sigma$ and $\lambda$ such that $|\Sigma|=1$. Additionally, assume that $f$ satisfies condition specified in Prop. 4.2, and $\Pi$ an MFM prior on $\mathcal{P}(\Theta)$ specified in Lemma 4.3. Then, as $n$ tends to infinity,

[TABLE]

in $p_{G_{0},f_{0}}$-probability for any positive constant $m<4/(4+5d)$.

The proof of Theorem 4.2 is straightforward using the result in Proposition 4.2 and analogous to the proof argument of Theorem 4.1; therefore, it is omitted. Note that, identical to the Gaussian kernel case, a similar contraction behavior also holds for the Laplace kernel with the Dirichlet Process Prior. The proof can be obtained similar to the MFM prior by invoking Lemma 5 of [37] instead of Lemma 4.3.

Remarks

(i) It is worth noting that compared to the well-specified setting, the posterior contraction upper bound obtained for Gaussian location mixtures remains the same slow logarithmic rate $(\log\log n/\log n)^{1/2}$. For Laplace mixtures, when the truth $f_{0}$ satisfies condition (P.5) with $\alpha\leq 1$, the posterior contraction upper bound obtained under misspecification remains a polynomial rate of the form $n^{-\gamma^{\prime}}$ modulo a logarithmic term. Due to misspecification there is a loss of a constant factor in the exponent $\gamma^{\prime}$, which is dependent on the shape of the kernel density as it is captured by the term $\tau(\alpha)$.

(ii) Although Gaussian mixtures have proved to be an asymptotically optimal density estimation device under suitable and mild conditions (cf. [18]), the results obtained in this section suggest that it is not a suitable choice for mixture modeling under model misspecification, even if the true $G_{0}$ has finite number of support points, if the primary interest is in the quality of model parameter estimates. Mixtures of heavy-tailed and ordinary smooth kernel densities such as the Laplace prove to be more amenable to efficient parameter estimation. Thus, the modeler is advised to select for $f$, say, a Laplace kernel over a supersmooth kernel such as Gaussian kernel, provided that condition (P.5) is valid.

(iii) It is interesting to consider the scenario where the true kernel $f_{0}$ happens to be a Gaussian kernel: if we use the either a well-specified or a misspecified Gaussian kernel to fit the data, the posterior contraction bound is the extremely slow $(\log\log n/\log n)^{1/2}$ accordingly to Theorem 4.1. This rate may be too slow to be practical. If the statistician is too impatient get to the truth $G_{0}$, because sample size $n$ is not sufficiently large, he may well decide to select a Laplace kernel $f$ instead. Despite the intentional misspecification, he might be comforted by the fact that the posterior distribution of $G$ contracts at an exponentially faster rate to a $G_{*}$ given by Theorem 4.2 for $\alpha=2$.

4.3 When $G_{*}$ has finite support

The source of the deterioriation in the statistical efficiency of parameter estimation under model mispecification is ultimately due to the increased complexity of the limiting point $G_{*}$. Even if the true $G_{0}$ has a finite number of support points, this is not the case for $G_{*}$ in general. Unfortunately, it is very difficult to gain concrete information about $G_{*}$ both in practice and in theory, due to the lack of knowledge about the true $p_{G_{0},f_{0}}$. When some precious information about $G_{*}$ is available, specifically, suppose that we happen to know $G_{*}$ has a bounded number of support points $k_{*}$ such that $k_{*}<\overline{k}$ for some known $\overline{k}$. Then it is possible to devise a new prior specification on the mixing measure $G$ so that one can gain a considerably improved posterior contraction rate toward $G_{*}$. We will show that it is possible to obtain the contraction rate of the order $(\log n/n)^{1/4}$ under $W_{2}$ metric — this is the same rate of posterior contraction one would get with overfitted mixtures in the well-specified regime.

In order to analyze the convergence rate of mixing measure under that setting of $k_{*}$, we introduce a relevant notion of integral Lipschitz property, which is a generalized form of the uniform Lipschitz property for the misspecification scenarios.

Definition 4.2**.**

*For any given $r\geq 1$, we say that the family of densities $f$ admits the

• integral Lipschitz* property up to the order $r$ with respect to two mixing measures $G_{0}$ and $G_{*}$ , if $f$ as a function of $\theta$ is differentiable up to the order $r$ and its partial derivatives with respect to $\theta$ satisfy the following inequality*

[TABLE]

for any $\gamma\in\mathbb{R}^{d}$ and for some positive constants $\delta$ independent of $x$ and $\theta_{1},\theta_{2}\in\Theta$. Here, $C(x)$ is some function such that ${\displaystyle\int C(x)\dfrac{p_{G_{0},f_{0}}(x)}{p_{G_{*}}(x)}\mathrm{d}x}<\infty$.

It is clear that when $f$ has integral Lipschitz property up to the order $r$, for some $r\geq 1$, with respect to $G_{0}$ and $G_{*}$, then it will admit uniform Lipschitz property up to the order $r$. We can verify that the first order intergral Lipschitz property is satisfied by many popular kernels, including location-scale Gaussian distribution and location-scale Cauchy distribution.

In the following we shall work with the MFM prior (7). Moreover,

• (M.0)

$q_{K}$ places positive masses on $K\in\left\{1,\ldots,\overline{k}\right\}$ and [math] mass elsewhere, where $\overline{k}\gg k_{*}$ is a fixed number.

Given that $k_{*}$ is finite, we obtain a key lower bound of weighted Hellinger distance in terms of the Wasserstein metric under strong identiability of $f$:

Proposition 4.3**.**

Assume that $f$ is second order identifiable and admits uniform integral Lipschitz property up to the second order. Then, for any $G\in\mathcal{O}_{\overline{k}}$, the following inequality holds

[TABLE]

The proof of Proposition 4.3 is in Section 7.2. Before stating the final theorem of this section, we will need following assumptions:

• (M.1)

The assumptions of Proposition 4.3 hold, i.e., $f$ is second order identifiable and admits uniform integral Lipschitz property up to the second order.

• (M.2)

There exists $\epsilon_{0}>0$ such that ${\displaystyle\int(p_{G_{0},f_{0}}(x))p_{G_{*}}(x)/p_{G}(x)d\mu(x)\leq M^{*}(\epsilon_{0})}$ whenever we have $W_{1}(G,G_{*})\leq\epsilon_{0}$ for any $G\in\mathcal{O}_{k_{*}}$ where $M^{*}(\epsilon_{0})$ depends only on $\epsilon_{0}$, $G_{*}$, $G_{0}$, and $\Theta$.

• (M.3)

The parameter $\gamma$ in Dirichlet distribution in MFM satisfies $\gamma<\overline{k}$. Additionally, the base distribution $H$ satisfies Assumption (P.2).

Theorem 4.3**.**

Assume $k_{0}<\infty$, and assumptions (M.0),(M.1),(M.2) and (M.3) hold. Then we have that,

[TABLE]

in $p_{G_{0},f_{0}}$-probability.

The proof of Theorem 4.3 is deferred to Section 9.5.

Further remarks

The above theorem raises a promising prospect for combating model misspecification, by having the modeler to fit the data to an underfitted mixture model $p_{G}$. Unfortunately, this theorem does not address this scenario, under which the limiting mixing measure would correspond to the KL minimizer

[TABLE]

for some $\overline{k}<\infty$, provided that this quantity exists (compare this with $G_{*}$ given in (10)). Due to the lack of convexity of the class of mixture densities with bounded number of mixture components, the theory developed in this section (tracing back to the work of [27]) is not applicable. Thus, posterior contraction behaviors in an underfitted mixture models remain an interesting open question.

5 Simulation studies

In this section we provide an illustration of the MTM algorithm’s behavior via a simple simulation study. Figures 5, 5, 5 and 5 illustrate the different stages in the application of MTM algorithm 1. In each figure, green dots denote the atoms in the set of ”remaining atoms” at each stage, with weights proportional to their sizes. Red dots denote the supporting atoms of the true mixing measure $G_{0}$. Black circles denote balls of radius $\omega_{n}$ around each of the ”remaining atoms”. Blue circles denote balls of radius $\frac{\omega_{n}}{4k_{0}}$ around the atoms of $G_{0}$.

Starting with an input measure $G$ represented in Fig. 5, the first stage of the algorithm (merge procedure, from line 1 to line 4) merges nearby atoms to produce $G^{\prime}$, which is represented by Fig. 5. There remains some atoms that carry very small mass, they are suitably truncated (via line 5 in the algorithm), and then merged accordingly (via line 6). Fig. 5 and Fig. 5 represent the outcome after these two steps of the algorithm. Observe how the atoms in each of the blue circles are merged to produced a reasonably accurate estimate of the corresponding atom of $G_{0}$. The number of such circles gives the correct number of the supporting atoms of $G_{0}$.

Next, we illustrate the performance of the MTM algorithm as it is applied to the samples from a Dirichlet process mixture, given the data generated by mixtures of three location Gaussian distributions:

[TABLE]

where $\mathcal{N}(\cdot|\mu,\Sigma)$ is the Gaussian distribution with mean vector $\mu$ and covariance matrix $\Sigma$. For simulation purposes, we consider the following four different settings ($n$ is the sample size):

Case A: $\mu_{1}^{0}=(0.8,0.8)$, $\mu_{2}^{0}=(0.8,-0.8)$, $\mu_{3}^{0}=(-0.8,0.8)$, $\Sigma^{0}=0.05I_{3}$, $n=500$. 2. 2.

Case B: $\mu_{1}^{0}=(0.8,0.8)$, $\mu_{2}^{0}=(0.8,-0.8)$, $\mu_{3}^{0}=(-0.8,0.8)$, $\Sigma^{0}=0.05I_{3}$, $n=1500$. 3. 3.

Case C: $\mu_{1}^{0}=(1.8,1.8)$, $\mu_{2}^{0}=(1.8,-1.8)$, $\mu_{3}^{0}=(-1.8,1.8)$, $\Sigma^{0}=0.05I_{3}$, $n=500$. 4. 4.

Case D: $\mu_{1}^{0}=(0.8,0.8)$, $\mu_{2}^{0}=(0.8,-0.8)$, $\mu_{3}^{0}=(-0.8,0.8)$, $\Sigma^{0}=0.01I_{3}$, $n=1500$.

Here, $I_{3}$ is the identity matrix of dimension $3$. Additionally, the weight vector for all these cases is chosen as $p^{0}=(p_{1}^{0},p_{2}^{0},p_{3}^{0})=(0.4,0.3,0.3)$.

As mentioned above, a Dirichlet process prior with an uniform prior base measure $H$ in the region $[-6,6]\times[-6,6]$, along with concentration parameter $\alpha=1$. This choice of prior enables us to sample significantly larger numbers of components of the mixing measure than the true number of three components.

It is known that the contraction rate of mixing measures under location Gaussian DPMM is $\tilde{C}(\log(n)^{-1/2})$ with respect to the Wasserstein-$2$ norm, for some constant $\tilde{C}$ which depends on $\Sigma^{0}$(the covariance matrix), the location parameters $\mu_{i}^{0}$ and the weights $p_{i}^{0}$ [37]. For our purpose, in order for $\omega_{n}$ to satisfy Equation (9), we may choose any $\omega_{n}$ as long as $\frac{\omega_{n}}{\log(n)^{-1/2}}\to\infty$. We selected \omega_{n}=\biggr{(}\frac{\log(\log(n))}{(\log(n))}\biggr{)}^{1/2} for all our applications of the MTM algorithm.

The MTM algorithm is provably consistent (in the asymptotic sense) for all chosen constants $c>0$. In practice for $n$ being fixed, the input $c$ to Algorithm 1 should be chosen so that $\frac{\tilde{C}}{(\log(\log(n)))^{1/2}}\leq c$. Moreover, for finite $n$ it is not expected that the posterior probability for $k=k_{0}$ is close to 1. However, for identifying the number of components the posterior mode provides a reasonable estimate. In particular, $(1-\sum_{i=1}^{3}\frac{c}{p_{i}^{0}})$ forms a useful lower bound on the posterior mass at the mode as identified in Equation (26). To identify $k=k_{0}$ consistently using the posterior mode safely, one needs to choose $c<c_{0}$, with $c_{0}$ satisfying $(1-\sum_{i=1}^{3}\frac{c_{0}}{p_{i}^{0}})>1/2$. The exact computation of the upper bound $c_{0}$ and the lower bound $\frac{\tilde{C}}{(\log(\log(n)))}$ for $c$ may be unrealistic but a reasonable estimate may be possible. Nonetheless, we simply considered a large range of $c$ and show there is a range where we can robustly identify the true number of components via the posterior mode.

For the DP mixture’s posterior computation, we make use of the non-conjugate split-merge sampler of Jain and Neal [25] with $(5,1,1,5)$ scheme, i.e., $5$ scans to reach the split launch state, $1$ split-merge move per iteration, $1$ Gibbs scan per iteration, and $5$ moves to reach the merge launch state. We run our experiments for two settings corresponding to sample sizes $500$ and $1500$. The sampler had 2000 burn-in iterations followed by 18000 sample iterations (a total 20000), with each 10th iteration being counted.

The experiments run for DP mixture-based sampler, followed by application of the MTM procedure for 4 different values of the tuning parameter $c$ in Algorithm 1, namely, for $c=0.45,0.5,0.55,1.0$. The proportional frequencies are plotted in Figure 5 and Figure 5 respectively, along with the proportional frequencies for DP mixture. The uniform base measure for the Dirichlet Process prior is chosen so as to enable easier creation of newer components in the split-merge scheme. As a consequence the DP mixture’s posterior yields quite bad results as far as the number of mixture components is concerned. However, even under that case, we can recover the true number of components by considering the mode of the frequency distribution after an application of the MTM algorithm on the posterior samples, with appropriate constant $c$. It is expected, however, that a large choice of $c$ would underestimate the number of components. This is also what is observed from the simulations, where the procedure breaks down when $c=1.0$.

We perform the experiments under four different settings of data populations. In particular, figure 5 consists of data generated from mixture of Gaussians with more widely spread location parameter values. In this case, it is expected that the convergence to the true number of components via Algorithm 1 will be faster for the posterior mode, in comparison to the situation where the location parameters are closer together. This is indeed what is observed in our simulations. The value of the covariance matrix $\Sigma^{0}$, on the other hand does not seem to noticeably affect the results. This is again expected, since the prior support $[-6,6]\times[-6,6]$ is quite large in comparison to the eigenvalues of the covariance matrix chosen.

6 Proofs of key results

In this section, we provide proofs for several key results in the paper.

6.1 Proof of Theorem 3.1

The proof of the theorem consists of two key parts. First, we recall a general framework for establishing posterior contraction of mixing measures. Then we proceed to apply this framework to analyze the specific setting of the MFM model.

6.1.1 General framework

To establish convergence rates of mixing measures under the setting of MFM, we utilize the general framework of posterior contraction of mixing measures under well-specified setting from [37]. To state such results formally, we will need to introduce several key definitions in harmony with the notations in our paper. Let $G$ be endowed with a prior distribution $\Pi$ on a measure space of discrete probability measures in $\overline{\mathcal{G}}(\Theta)$. Fix $G_{0}\in\mathcal{P}(\Theta)$. For any set $\mathcal{S}\subset\overline{\mathcal{G}}(\Theta)$, we define the Hellinger information of the $W_{1}$ metric for subset $\mathcal{S}$ by the following function

[TABLE]

Note that, the choice of first order Wasserstein metric in the above formulation is due to the lower bound of Hellinger distance between mixing densities in terms of first order Wasserstein distance between their corresponding mixing measures in (1). Now, for any mixing measure $G_{1}\in\overline{\mathcal{G}}(\Theta)$ and $r>0$, we define a Wasserstein ball centered at $G_{1}$ under $W_{1}$ metric as follows

[TABLE]

Furthermore, for any $M>0$, we define a Kullback-Leibler neighborhood of $G_{0}$ by

[TABLE]

For the proof of Theorem 3.1, we use a straightforward extension of Theorem 4 in [37], adapted to the setting in this work.

Theorem 6.1**.**

Fix $G_{0}\in\overline{\mathcal{G}}$. Assume the following:

• (a)

The family of likelihood functions is finitely identifiable and satisfies $h(f(x|\theta_{i}),f(x|\theta_{j}^{\prime}))\leq C_{1}\|\theta_{i}-\theta_{j}^{\prime}\|^{\alpha}$ for any $\theta_{i},\theta_{j}^{\prime}\in\Theta$, for some constants $C_{1}>0$, $\alpha\geq 1$.

• (b)

There is a sequence of sets $\mathcal{G}_{n}\subset\overline{\mathcal{G}}$ for which

[TABLE]

• (c)

There is a sequence $\epsilon_{n}\to 0$ such that $n\epsilon_{n}^{2}$ is bounded away from 0 or tending to infinity, a constant $M>0$ sufficiently large, and a sequence $M_{n}$ such that

[TABLE]

Then, we have that $\Pi(G\in\overline{\mathcal{G}}:\ W_{1}(G,G_{0})\geq M_{n}\epsilon_{n}|X_{1},\ldots,X_{n})\to 0$ in $P_{G_{0}}$-probability.

6.1.2 Posterior contraction under MFM

Now, we apply the above result to establish the convergence rate of mixing measure under a well-specified MFM model. The constant $M$ for part (c) of Theorem 6.1 is chosen later. Also, let $\epsilon_{n}:=\overline{M}(\log n/n)^{1/2}$ where $\overline{M}$ is a sufficiently large constant that will be chosen later. Note that it is enough to show, \Pi\biggr{(}G\in\mathcal{G}(\Theta):W_{1}(G,G_{0})\gtrsim\dfrac{(\log n)^{1/2}}{n^{1/2}}\biggr{|}X_{1},\ldots,X_{n}\biggr{)}\to 0, since $\Pi(G\in\overline{\mathcal{G}}(\Theta)\setminus\mathcal{G}(\Theta)|X_{1},\ldots,X_{n})=0$.

With $\epsilon_{n}$ chosen as above, we denote $A_{n}:=\Pi(G\in\mathcal{G}(\Theta):W_{1}(G,G_{0})\gtrsim\epsilon_{n}|X_{1},\ldots,X_{n})$. It is clear that

[TABLE]

Now, we divide our proof into the following key steps

Step 1:

$\Pi(K=k_{0}|X_{1},\ldots,X_{n})\to 1$ a.s. $P_{G_{0}}$. As the model is identifiable, this result is the direct application of Doob’s consistency theorem [7].

Step 2:

P_{G_{0}}\biggr{(}\Pi(G\in\mathcal{O}_{k_{0}}(\Theta):W_{1}(G,G_{0})\gtrsim\epsilon_{n}|X_{1},\ldots,X_{n})\biggr{)}\to 0 as $n\to\infty$. The proof of this result is the application of Theorem 6.1. In fact, as we focus on the posterior contraction of $G_{0}$ from $G\in\mathcal{O}_{k_{0}}(\Theta)$, we denote the prior on $G\in\mathcal{O}_{k_{0}}(\Theta)$ to be $\Pi=H\times Q$ where $Q\overset{d}{=}\text{Dir}(\gamma/k_{0},\ldots,\gamma/k_{0})$. Now, we claim that

[TABLE]

where $c_{H}>0$ and $\gamma^{\prime}>0$ are some positive constants and $\epsilon$ is sufficiently small. Assume that claim (18) is given at the moment. To facilitate the discussion, we further divide Step 2 into two small steps.

Step 2.1:

To obtain the bound for $\Pi(B_{K}(\epsilon_{n},M))$, we utilize the result from [53] to bound KL divergence and squared KL divergence. In particular, from Theorem 5 of [53], if $p$ and $q$ are two densities such that $2h^{2}(p,q)\leq\epsilon^{2}$ and ${\int p^{2}/q\leq M^{2}}$ then we obtain that $K(p,q)\lesssim\epsilon^{2}\log(M/\epsilon)$ and $K_{2}(p,q)\lesssim\epsilon^{2}(\log(M/\epsilon))^{2}$ where the constants in these bounds are universal.

Now, since $f$ admits Lipschitz continuity up to the first order, we achieve that $h^{2}(p_{G},p_{G_{0}})\leq C_{1}W_{1}(G,G_{0})$ for any $G\in\mathcal{O}_{k_{0}}(\Theta)$ where $C_{1}$ is a positive constant depending only on $\Theta$. Now, for any $G\in\mathcal{O}_{k_{0}}(\Theta)$ such that $W_{1}(G,G_{0})\leq C\epsilon_{n}^{2}$ where $C<\epsilon_{0}$ is a sufficiently small constant to be chosen later, the previous bound implies that $h^{2}(p_{G},p_{G_{0}})\leq C_{1}C\epsilon_{n}^{2}$. Since $C\epsilon_{n}^{2}\leq\epsilon_{0}$ for all $n$ sufficiently large, we also have that ${\displaystyle\int(p_{G_{0}}(x))^{2}/p_{G}(x)\mathrm{d}\mu(x)\leq M(\epsilon_{0})}$ according to assumption (P.3). Combining all the previous results, we achieve that

[TABLE]

when $\overline{M}$ is sufficiently large. Define $M:=\sqrt{M(\epsilon_{0})/CC_{1}}$. Therefore, we have

[TABLE]

For any $G=\sum_{i=1}^{k_{0}}{p_{i}\delta_{\theta_{i}}}$ such that $\|\theta_{i}-\theta_{i}^{0}\|\leq\overline{\epsilon}$ and $|p_{i}-p_{i}^{0}|\leq\overline{\epsilon}/(k_{0}\text{Diam}(\Theta))$ for any $1\leq i\leq k_{0}$ and sufficiently small $\overline{\epsilon}>0$, we can check that

[TABLE]

Hence, by choosing universal constant $C$ such that $C\epsilon_{n}^{2}\leq\overline{\epsilon}$, we would have that

[TABLE]

where the last inequality is due to the results from claim (18).

Step 2.2:

To apply the posterior contraction rate result of Theorem 6.1, we choose $\mathcal{G}_{n}=\mathcal{O}_{k_{0}}(\Theta)$ for all $n$. Now, it is clear that $\Pi(\overline{\mathcal{G}}\backslash\mathcal{G}_{n})=0$. Therefore, condition (15) is obviously satisfied. Additionally, by means of Lemma 4 in [37], we can check that condition (14) is satisfied with our choice of $\epsilon_{n}$ as $\overline{M}$ is sufficiently large. For condition (16), from the bound of KL neighborhood in (19), we find that

[TABLE]

Since $f$ is first order identifiable and admits uniform Lipschitz property up to the first order, according to (1), we obtain that $\Psi_{\mathcal{G}_{n}}(r)\gtrsim Cr^{2}$ for any $r>0$ where $C$ is some positive constant that depends only on $G_{0}$ and $\Theta$. Therefore, we have

[TABLE]

for any $j\geq M_{n}$. By choosing $M_{n}$ such that $M_{n}^{2}\geq 32(c_{H}+\gamma)/C$, it is clear that condition (16) is satisfied.

For condition (17), combining with the above bound of $\Psi_{\mathcal{G}_{n}}(r)$, we would have that

[TABLE]

as long as $M_{n}$ is chosen such that $M_{n}^{2}\geq\frac{32\overline{M}^{2}}{C}$. Therefore, condition (17) is satisfied. As a consequence, we achieve the conclusion of the theorem.

Proof of claim (18)

According to the formulation of Dirichlet distribution, we obtain that

[TABLE]

Now, for $\gamma/k_{0}\leq 1$, equation (20) can be re-written as:

[TABLE]

Here, the first inequality in the above display follows from the fact that $1-\sum\limits_{i=1}^{k_{0}-1}p_{i}\leq 1$ while the second inequality is due to direct integration and the fact that $\epsilon$ is sufficiently small.

On the other hand, for $\gamma/k_{0}>1$ , we can rewrite equation (20) as

[TABLE]

where the above inequality follows due to the fact that $p_{i}^{0}>0$ for all $i\in\{1,\dots,k_{0}\}$ and that for sufficiently small $\epsilon>0$ such that $|p_{i}-p_{i}^{0}|\leq\epsilon$ for all $i\in\{1,\dots,k_{0}-1\}$, we have $p_{k_{0}}\geq p_{k_{0}}^{0}/2$. Therefore, the lower bound for the Dirichlet distribution $Q$ follows automatically from the results of these two separate conditions of $\gamma/k_{0}\leq 1$ and $\gamma/k_{0}>1$.

On the other hand, to show the bounds for $H$, we note that $\Theta\subset\mathbb{R}^{d}$. Suppose $B(\theta,\epsilon):=\{\theta^{\prime}\subset\Theta:\|\theta-\theta^{\prime}\|\leq\epsilon\}$ denotes the $\ell_{2}$ ball in $\Theta\cup\mathbb{R}^{d}$ around $\theta$, with radius $\epsilon$. Then it can be seen that

[TABLE]

since $B(\theta_{i}^{0},\epsilon)$ are disjoint for all $i\leq k_{0}$ , for sufficiently small $\epsilon>0$. Here, $\mu(A)$ denotes the $d$-dimensional Lebesgue measure of the set $A\subset\Theta$ and $g$ is the density function of $H$ based on Assumption (P.2). Using the fact that $\epsilon^{d}\lesssim\mu(B(\theta^{\prime},\epsilon))\lesssim\epsilon^{d}$ and the condition that $H$ is approximately uniform in Assumption (P.2), the remainder of the claim follows.

6.2 Posterior consistency of Merge-Truncate-Merge algorithm

The goal of this section is to both deliver a proof of Theorem 3.2 and clarify the role played by each of the steps of the MTM algorithm.

6.2.1 Probabilistic scheme for merging atoms

The first step of MTM algorithm comprises of lines from 1 to 3 in Algorithm 1. It describes a probabilistic scheme for merging atoms from an input measure $G$. Recall that $G$ is a sample from the posterior distribution of a mixing measure which is assumed to be relatively close to the true $G_{0}$, per Eq. (9). To simplify notations within this subsection we shall remove subscript $n$ in $\omega_{n}$ in (9), namely, we will not incorporate the randomness of data in the results in this subsection.

In that regard, suppose that we have a measure $G=\sum_{j}p_{j}\delta_{\theta_{j}}\in\overline{\mathcal{G}}(\Theta)$ such that $W_{r}(G,G_{0})\leq\delta\omega$ for some $r\geq 1$. Here, $\delta,\omega$ are sufficiently small such that the following two properties hold:

• (B.1)

$\omega<\min\{(p_{\min}^{0}/2)^{1/r},\min_{u\neq v}\frac{\|\theta_{u}^{0}-\theta_{v}^{0}\|}{8}\}$.

• (B.2)

$\sqrt{\delta}<p_{\min}^{0}/(2k_{0})$, where $p_{\min}^{0}:=\min_{i=1}^{k_{0}}p_{i}^{0}$.

Denote by $\mathcal{A}(G)$ the set of atoms corresponding to any mixing measure $G$. For a given $G$ and $\omega$, let $g_{\omega,G}$ be the set of all discrete measures which collect the atoms from $G$ such that all their atoms spaced apart by a distance at least $\omega$:

[TABLE]

Note in this definition that any $G^{\prime}\in g_{\omega,G}$ must have finite number of atoms, because $\Theta$ is compact.

The first merge step in the MTM algorithm is motivated by the following result, which establishes the existence of a probabilistic procedure that transform $G$ into another measure $G^{\prime}\in g_{\omega,G}$ that possesses some useful properties, namely, the supporting atoms of $G^{\prime}$ are well-separated from one another, while $G^{\prime}$ remains sufficiently close to $G_{0}$ in the sense of a Wasserstein metric.

Lemma 6.1**.**

Assume that $W_{r}(G,G_{0})\leq\delta\omega$ for some $r\geq 1$ where $\omega,\delta$ satisfy condition (B.1) and (B.2). Then, there exists a probabilistic scheme which transform $G$ into a $G^{\prime}=\sum_{j=1}^{k}p^{\prime}_{j}\delta_{\theta^{\prime}_{j}}$ such that $k\geq k_{0}$ and the following holds:

[TABLE]

Here, $P$ is the probability measure associated with the probabilistic scheme and the conditions (G.1) and (G.2) stand for

1. (G.1)

$G^{\prime}\in g_{\omega,G}$* and $W_{r}(G^{\prime},G_{0})\leq(k_{0}+2)\sqrt{\delta}\omega$.* 2. (G.2)

For each $i=1,\ldots,k_{0}$ there is an index $j$ for an atom of $G^{\prime}$ for which $|p_{j}-p_{i}^{0}|\leq\delta^{r/2}$ and $\|\theta^{\prime}_{j}-\theta_{i}^{0}\|\leq\sqrt{\delta}\omega$.

Proof.

The probabilistic scheme is the first merge step described in the MTM algorithm. We recall it in the following

Reorder the indices of components $\{\theta_{1},\dots,\theta_{|G|}\}$ by simple random sampling without replacement (SRSWOR) with corresponding weights $\{p_{1},\dots,p_{|G|}\}$. 2. 2.

Let $\tau_{1},\dots,\tau_{|G|}$ denote the new indices, and set $\mathcal{E}=\{\tau_{j}\}_{j}$ as the existing set of atoms. 3. 3.

Sequentially for each index $\tau_{j}$, if there exists an index $\tau_{i}<\tau_{j}$ such that $\|\theta_{\tau_{i}}-\theta_{\tau_{j}}\|\leq\omega$, we perform the following updates

• •

update $p_{\tau_{i}}=p_{\tau_{i}}+p_{\tau_{j}}$.

• •

update $\mathcal{E}$ by removing index $\tau_{j}$ from $\mathcal{E}$. 4. 4.

Set $G^{\prime}=\sum_{j:\ \tau_{j}\in\mathcal{E}}p_{\tau_{j}}\delta_{\theta_{\tau_{j}}}$.

The proof consists of two main steps. First, we shows that every atom of $G_{0}$ lies in a $\sqrt{\delta}\omega$ neighborhood of a unique atom of $G^{\prime}=\sum_{i=1}^{k}p^{\prime}_{i}\delta_{\theta^{\prime}_{i}}$ having large mass, with high probability. This will allows us to deduce that $G^{\prime}$ satisfies (G.2) with a high probability. Next, we shall show that

[TABLE]

to conclude the lemma. Note that by the nature of construction it automatically holds that $G^{\prime}\in g_{\omega,G}$.

Step 1:

Let $P(B|G)$ be the probability of an event $B$ under the SRSWOR scheme used above, conditioned the mixing measure $G$. Furthermore, let $G(A)$ denote the mass assigned to the set $A\subset\Theta$ by measure $G$. Thus, for a given $\epsilon>0$

[TABLE]

for any $\theta\in\Theta$. Now, the amount of mass transfer between $\theta_{i}^{0}$ and those atoms of $G$ residing in $\mathbb{B}(\theta_{i}^{0},\epsilon\omega)^{\mathsf{c}}$ is at least $|p_{i}^{0}-G(\mathbb{B}(\theta_{i}^{0},\epsilon\omega))|$. Therefore, as $W_{r}(G,G_{0})\leq\delta\omega$,

[TABLE]

for any index $i\in\{1,\dots,k_{0}\}$ and for any $2\geq\epsilon>0$. The upper bound of $2$ arises from the consideration of selecting disjoint balls combined with the fact that $\omega<\min_{u\neq v}\frac{\|\theta_{u}^{0}-\theta_{v}^{0}\|}{4}$. It leads to the following inequalities

[TABLE]

Since $\omega<\min_{u\neq v}\frac{\|\theta_{u}^{0}-\theta_{v}^{0}\|}{4}$, based on the standard union bound, the following inequality holds

[TABLE]

The last inequality in the above display holds because $1\geq p_{\min}^{0}>2k_{0}\sqrt{\delta}$. Now, combining Equations (23) with (22), for specific choice of $\epsilon=\sqrt{\delta}$, we get that

[TABLE]

Divide $\Theta$ into disjoint subsets $\Theta=A_{1}\cup\ldots\cup A_{k_{0}+1}$, where $A_{i}=\mathbb{B}(\theta_{i}^{0},\sqrt{\delta}\omega)$ for all $i\leq k_{0}$, and $A_{k_{0}+1}=(\cup_{i=1}^{k_{0}}\mathbb{B}(\theta_{i}^{0},\sqrt{\delta}\omega))^{c}$. For each $i=1,\ldots,k_{0}$, let $E^{i}$ denote the event that an atom of $G^{\prime}$ resides in $A_{i}$. The probabilistic scheme for the selection of atoms for $G^{\prime}$ from the those of $G$ (via random sampling without replacement) will pick an atom from $A_{i}$ and gives it a lower index than one from $A_{k_{0}+1}$ with probability

[TABLE]

Moreover, if $G^{\prime}\in E^{i}$ and $\theta^{\prime}_{j}\in\mathcal{A}(G^{\prime})$ such that $\|\theta^{\prime}_{j}-\theta_{i}^{0}\|\leq\sqrt{\delta}\omega$, then

[TABLE]

Thus $G^{\prime}\in E^{i}$ satisfies (G.2).

This entails that P(\{G^{\prime}:G^{\prime}\text{ satisfies (G.2)}\}|G)\geq P(\cap_{i=1}^{i_{0}}E^{i}|G)\geq 1-\biggr{(}\sum_{i=1}^{k_{0}}\frac{\delta^{r/2}}{p_{i}^{0}}\biggr{)}, which concludes the first proof step.

An useful fact to be used later is that if $\theta_{i}^{0}\in\mathbb{B}(\theta_{j}^{\prime},\sqrt{\delta}\omega)$ for some $j\leq k$, when $\theta_{j}^{\prime}\in\mathcal{A}(G^{\prime})$, $G^{\prime}\in E^{i}$ , then $G(\mathbb{B}(\theta_{j},\omega))\geq\omega^{r}$. Indeed, suppose that this claim does not hold, then by the definition of Wasserstein metric, we find that

[TABLE]

which is a contradiction as we have $p_{\min}^{0}\geq 2k_{0}\sqrt{\delta}$ and $\omega^{r}<p_{i}^{0}/2$.

Step 2:

To establish (21) it suffices to assume that $G^{\prime}=\sum_{j}p^{\prime}_{j}\delta_{\theta^{\prime}_{j}}\in E^{i}$ satisfies that for every $i=1,\ldots,k_{0}$, $\|\theta^{\prime}_{i}-\theta_{i}^{0}\|\leq\sqrt{\delta}\omega$, and $|p_{i}^{0}-G(\mathbb{B}(\theta_{i}^{0},\sqrt{\delta}\omega))|\leq\delta^{r/2}$. Then, we have

[TABLE]

The above result leads to $G^{\prime}((\cup_{i=1}^{k_{0}}\mathbb{B}(\theta^{\prime}_{i},\omega))^{c})\leq k_{0}\delta^{r/2}$.

Now, we construct an measure $\tilde{G}=\sum_{l}\tilde{p}_{l}\delta_{\phi_{l}}$ from $G^{\prime}$ and $G$, by ”de-merging” all atoms of $G^{\prime}$ except for its first $k_{0}$ atoms. Specifically, for indices $l\leq k_{0}$, simply take $\tilde{p}_{l}=p^{\prime}_{l}$ and $\phi_{l}=\theta^{\prime}_{l}$. Additionally, if index $l>k_{0}$ is such that $\|\theta_{l}-\theta^{\prime}_{i}\|>\omega$ for all $i\leq k_{0}$, then $\phi_{l}=\theta_{l}$ and $\tilde{p}_{l}=p_{l}$. Otherwise, let $\tilde{p}_{l}=0$. By the triangle inequality with Wasserstein metric,

[TABLE]

The second inequality above holds because $\sum_{l=1}^{k_{0}}\tilde{p}_{l}=\sum_{j=1}^{k_{0}}p^{\prime}_{j}\geq 1-k_{0}\delta^{r/2}$. So, there exists a coupling of $G^{\prime}$ and $\tilde{G}$ such that any mass transfer occurs between atoms located at most $\omega$ in distance from each other. Moreover, the coupling can be so obtained that the total mass travelling a non-zero distance is bounded above by $k_{0}\delta^{r/2}$.

It remains to obtain a suitable upper bound for $W_{r}^{r}(\tilde{G},G_{0})$. From the definition of Wasserstein metric, we can write

[TABLE]

where $\mathcal{Q}(\boldsymbol{\tilde{p}},\boldsymbol{p^{0}})$ is the set of all possible couplings between $\boldsymbol{\tilde{p}}=(\tilde{p}_{1},\ldots,\tilde{p}_{|G|})$ and $\boldsymbol{p^{0}}=(p_{1}^{0},\ldots,p_{k_{0}}^{0})$. Now, we consider a coupling $\boldsymbol{q}\in\mathcal{Q}(\boldsymbol{\tilde{p}},\boldsymbol{p^{0}})$ such that $q_{ii}=\min\{p_{i}^{0},\tilde{p}_{i}\}$ for any $i$. Then, the following inequalities hold

[TABLE]

Therefore, following Equation (24), we have,

[TABLE]

As a consequence, we achieve the conclusion of the lemma. ∎

6.2.2 Truncate-merge scheme

In the previous subsection we studied properties of the first stage of the MTM algorithm, which is applied an arbitrary discrete measure $G$ that is sufficiently close to $G_{0}$ under Wasserstein metric, namely, $W_{r}(G,G_{0})\leq\delta\omega$ for some small quantities $\delta>0$ and $\omega>0$. The next stage of the MTM algorithm comprises of lines 4 to 7 in the algorithm’s description. It is applied to a measure $G^{\prime}$, which is the outcome of the algorithm’s first stage. Denote $G^{\prime}=\sum_{j=1}^{k}p^{\prime}_{j}\delta_{\theta^{\prime}_{j}}$ where $k\geq k_{0}$. As a consequence of Lemma 6.1, $G^{\prime}$ satisfies two important properties (G.1) and (G.2), which are to be restated here for the reader’s convenience.

1. (G.1)

$G^{\prime}\in g_{\omega,G}$ and $W_{r}(G^{\prime},G_{0})\leq\delta^{\prime}\omega$, where $\delta^{\prime}=(k_{0}+2)\sqrt{\delta}$. 2. (G.2)

For each $i\leq k_{0}$, there exists an (unique) atom of $G^{\prime}$, which is relabeled $\theta^{\prime}_{i}$ so that $|\theta_{i}^{0}-\theta^{\prime}_{i}\|\leq\sqrt{\delta}\omega\leq\omega p_{\min}^{0}/(2k_{0})\leq\omega/(2k_{0})$.

By definition, $G^{\prime}\in g_{\omega,G}$ implies that its atoms are well-separated, namely, for any $1\leq i<j\leq k$, $\|\theta_{i}^{\prime}-\theta_{j}^{\prime}\|\geq\omega$. Assuming slightly stronger conditions on the two quantities $\omega$ and $\delta$, we can say more about the structure of $G^{\prime}$, which turns out to be very useful in identifying the true number of atoms $k_{0}$ of $G_{0}$ via a truncation procedure.

1. (B.3)

$\omega<\frac{7p_{\min}^{0}\min_{u\neq v}\|\theta_{u}^{0}-\theta_{v}^{0}\|}{16}$. 2. (B.4)

$\sqrt{\delta}<p_{\min}^{0}/(2k_{0}(k_{0}+2))$, where $p_{\min}^{0}:=\min_{i=1}^{k_{0}}p_{i}^{0}$.

Lemma 6.2**.**

Suppose that $\omega$ and $\delta$ satisfy conditions (B.1), (B.3) and (B.4). Then for any $G^{\prime}$ satisfying properties (G.1) and (G.2), the following hold.

1. (a)

For each $1\leq i\neq j\leq k_{0}$, we obtain that $(p_{j}^{\prime})^{1/r}\|\theta_{i}^{\prime}-\theta_{j}^{\prime}\|>\omega$. 2. (b)

For each $j>k_{0}$, we find that $\min_{1\leq i\leq k_{0}}(p_{j}^{\prime})^{1/r}\|\theta_{i}^{\prime}-\theta_{j}^{\prime}\|\leq\omega$.

Proof.

To show (a), note for any $i,j\leq k_{0}$

[TABLE]

where the first inequality follows from triangle inequality, the second inequality is due to the hypothesis with $G^{\prime}$, and the third inequality is due to (B.1).

By the definition of Wasserstein distances, for mass transport to be achieved between $G^{\prime}$ and $G_{0}$, an amount of mass at least $|p_{i}^{0}-p^{\prime}_{i}|$ should be transported from atom $\theta_{i}^{0}$ of $G_{0}$ to an atom of $G^{\prime}$ other than $\theta^{\prime}_{i}$. Hence, for any $i\leq k_{0}$, $|p_{i}^{0}-p^{\prime}_{i}|(\omega-\|\theta^{\prime}_{i}-\theta_{i}^{0}\|)^{r}\leq W_{r}^{r}(G^{\prime},G_{0})\leq(\delta^{\prime}\omega)^{r}$. Invoking the hypothesis with $G^{\prime}$, these inequalities lead to $|p_{i}^{0}-p^{\prime}_{i}|^{1/r}\leq 2\delta^{\prime}$. Combining with the condition $\sqrt{\delta}<\frac{p_{i}^{0}}{2k_{0}(k_{0}+2)}$, the above inequality leads to $p^{\prime}_{i}>p_{i}^{0}-\frac{(p_{i}^{0})^{2}}{2k_{0}^{2}}\geq\frac{p_{i}^{0}}{2}$. Combining this with Equation (25) and (B.3) to conclude.

Turning to part (b), suppose for some $j>k_{0}$, we have $(p^{\prime}_{j})^{1/r}\|\theta^{\prime}_{i}-\theta^{\prime}_{j}\|>\omega$ for all $i\leq k_{0}$. Then by triangle inequality and the properties of $G^{\prime}$, we find that

[TABLE]

Applying the triangle inequality again, $\|\theta_{i}^{0}-\theta^{\prime}_{j}\|\geq\|\theta^{\prime}_{j}-\theta^{\prime}_{i}\|-\|\theta_{i}^{0}-\theta^{\prime}_{i}\|\geq\omega(1-\frac{1}{2k_{0}})\geq\frac{\omega}{2}$. Combining the two preceeding bounds, we get $2\|\theta_{j}^{\prime}-\theta_{i}^{0}\|>\omega/{p^{\prime}_{j}}^{1/r}$ for all $i\leq k_{0}$.

Now, since $W_{r}(G^{\prime},G_{0})\leq\delta^{\prime}\omega$, we can find a coupling $\boldsymbol{q}\in\mathcal{Q}(\boldsymbol{p^{\prime}},\boldsymbol{p}^{0})$ between $\boldsymbol{p^{\prime}}=(p^{\prime}_{1},\ldots,p^{\prime}_{k})$ and $\boldsymbol{p^{0}}=(p_{1},\ldots,p_{k_{0}}^{0})$ such that $\sum_{ij}q_{ij}\|\theta^{\prime}_{j}-\theta_{i}^{0}\|^{r}\leq(\delta^{\prime}\omega)^{r}$. However, based on the previous inequalities, we have for the given index $j$

[TABLE]

which is a contradiction as $\delta^{\prime}<1/2$ due to condition (B.4). This concludes the proof of the lemma.

∎

6.2.3 Proof of Theorem 3.2

Now we are ready for the proof of this theorem. It suffices to prove for the case constant $c=1$.

Proof of part (a):

Recall that we are given a (random) mixing measure for which the following holds: for each fixed $\epsilon>0$ and $\delta>0$, as $n\rightarrow\infty$, there holds

[TABLE]

Choose $\delta$ sufficiently small, and as $n$ gets large $\omega_{n}$ also becomes so small that all conditions (B.1–4) in the preceeding sections are satisfied. Then, we can appeal to Lemma 6.1 to obtain that, measure $G^{\prime}$ as produced in the probablistic merge stage of the MTM algorithm also admits a posterior contraction toward $G_{0}$, in the sense that as $n\rightarrow\infty$

[TABLE]

Since this holds for any $\delta>0$, we deduce that the posterior probability \Pi\biggr{(}W_{r}(G^{\prime},G_{0})\leq(k_{0}+2)\sqrt{\delta}\omega_{n}\big{|}(X_{1},\dots,X_{n})\biggr{)}\rightarrow 1 in $P_{G_{0}}$ probability.

Suppose that $G^{\prime}$ satisfies both conditions (G.1) and (G.2) (per Lemma 6.1), then it can be verified that if the atoms of $G^{\prime}$ are arranged in descending order of their masses, then each of the top $k_{0}$ atoms of $G^{\prime}$ lie in an $\frac{\omega_{n}}{2k_{0}}$- ball around an atom of $G_{0}$. Specifically, using the representation $G^{\prime}=\sum_{i=1}^{k_{n}}q_{i}\delta_{\phi_{i}}$, where $q_{1}\geq\dots\geq q_{k_{n}}$, we have that $\|\theta_{i}^{0}-\phi_{i}\|\leq\frac{\omega_{n}}{2k_{0}}$ and $|q_{i}-p_{i}^{0}|\geq\delta^{r/2}$ for all $i\in\{1,\ldots,k_{0}\}$.

Recall that $G^{\prime}$ is fed into the second stage, the truncate-merge procedure, of the MTM algorithm. Note that $|q_{i}-p_{i}^{0}|\leq\delta^{r/2}$ implies $q_{i}>p_{i}^{0}-\delta^{r/2}>p_{i}^{0}/2>\omega_{n}^{r}$ for $n$ sufficiently large. By Lemma  6.2 that for each $j>k_{0}$, $\min_{i\leq k_{0}}(q_{j})^{1/r}\|\phi_{i}-\phi_{j}\|\leq\omega_{n}$, but for each $i,j\leq k_{0},i\neq j$, $(q_{j})^{1/r}\|\phi_{i}-\phi_{j}\|\geq\omega_{n}$. Following the definition of $\tilde{k}=|\mathcal{A}|$, we deduce that $\tilde{k}=k_{0}$. The final step is to coat this guarantee with a probability statement, due to the fact that $G^{\prime}$ is random given $G$,

[TABLE]

Let $\delta\rightarrow 0$ to conclude the proof of part (a).

Proof of part (b):

The proof boils down to showing that the reassignment of mass as in the second stage of the MTM Algorithm only increases the Wasserstein distance by a constant factor. Denote by $\boldsymbol{p^{0}}=(p_{1}^{0},\ldots,p_{k_{0}}^{0})$ and $\boldsymbol{q}=(q_{1},\ldots,q_{k_{n}})$ the weight vectors of $G_{0}$ and $G^{\prime}$ respectively. Suppose that $W_{r}(G^{\prime},G_{0})\leq(k_{0}+2)\sqrt{\delta}\omega_{n}$ as before. So can find a coupling $\boldsymbol{f}\in\mathcal{Q}(\boldsymbol{p^{0}},\boldsymbol{q})$ such that

[TABLE]

Define the set $V_{i,n}:=\{\theta\in\Theta:\|\theta-\theta_{i}^{0}\|\leq\min_{u\neq v}\frac{\|\theta_{u}^{0}-\theta_{v}^{0}\|}{2}-\frac{\omega_{n}}{2k_{0}}\}$. Furthermore, the following inequalities hold

[TABLE]

for all $i\leq k_{0}$ and $j\not\in V_{i,n}$, because $\omega_{n}$ satisfies assumption (B.1). Therefore, we find that

[TABLE]

Notice that if $j\in V_{i,n}$, the second stage of the MTM Algorithm assigns the mass corresponding to atom $j$ of $G^{\prime}$ to atom $i$ of $\widetilde{G}$. We can assume henceforth that $\widetilde{G}$ is such that $|\mathcal{A}(\widetilde{G})|=k_{0}$ as a result of the proof of part (a) of this theorem.

Since the sets $V_{i,n}$ are disjoint, this assignment is unique. It follows that we can find $\boldsymbol{f}^{\prime}\in\mathcal{Q}(\boldsymbol{p_{0}},\boldsymbol{q^{\prime}})$ such that

[TABLE]

where $\boldsymbol{q^{\prime}}=(q^{\prime}_{1},\ldots,q^{\prime}_{k_{0}})$ is the weight vector of $\widetilde{G}$. The first inequality above follows, since $\|\phi_{i}-\theta_{i}^{0}\|\leq\|\phi_{j}-\theta_{i}^{0}\|$ for all pairs $i,j$ with $i\leq k_{0}$, with strict inequality for $i\neq j$. To obtain the conclusion for the second inequality in (6.2.3), we note that $p_{i}^{0}=\sum_{j}f_{ij}=\sum_{j}f^{\prime}_{ij}$. Therefore, $f^{\prime}_{ii}=\sum_{j}f_{ij}-\sum_{j\neq i}f^{\prime}_{ij}$. Then, if $\phi_{j}$ is an atom of $G^{\prime}$, for any $j\neq i$, we have $\|\theta_{i}^{0}-\phi_{i}\|\leq\|\theta_{i}^{0}-\phi_{j}\|$. Hence, we find that

[TABLE]

By the nature of construction $\mathcal{A}(\widetilde{G})\subset\mathcal{A}(G^{\prime})$. Using the two parts of Equation (6.2.3), we obtain that

$W_{r}(\widetilde{G},G_{0})\leq\left(1+4\left(\frac{2\text{Diam}(\Theta)}{\min_{i,l}\|\theta_{i}^{0}-\theta_{l}^{0}\|}\right)^{r}\right)^{1/r}((k_{0}+2)\sqrt{\delta}\omega_{n})$.

The full probability statement is

[TABLE]

where $C=\left(1+4\left(\frac{2\text{Diam}(\Theta)}{\min_{i,l}\|\theta_{i}^{0}-\theta_{l}^{0}\|}\right)^{r}\right)^{1/r}(k_{0}+2)$ is a constant dependent on $G_{0}$ and $\Theta$. Finally, letting $\delta\rightarrow 0$ we obtain the desired conclusion for part (b).

6.3 Proof of Lemma 4.3

To simplify the proof argument, we specifically assume that $G_{*}$ is a discrete mixture. The proof argument for other settings of $G_{*}$ is similar and is omitted. Now, we consider an $\epsilon>0$ maximal packing set of parameter space $\Theta$. It leads to a $D-$partition $(S_{1},\dots,S_{D})$ of $\Theta$ such that $\text{Diam}(S_{i})\leq 2\epsilon$ for all $1\leq i\leq D$. Choose $\epsilon$ to be sufficiently small such that $D>\gamma$.

For mixing measures $G=\sum_{i}p_{i}\delta_{\theta_{i}}$ and $G_{*}=\sum_{i=1}^{\infty}p_{i}^{*}\delta_{\theta_{i}^{*}}$, we denote $G(S_{i}):=\sum_{i:\theta_{i}\in S_{i}}p_{i}$ and $G_{*}(S_{i})=\sum_{i:\theta_{i}^{*}\in S_{i}}p_{i}^{*}$. Invoking the detailed formulation of Wasserstein metric, we can check that

[TABLE]

Equipped with the above inequality, the following inequality holds

[TABLE]

For any positive constant $A$, we find that

[TABLE]

where $B$ stands for the event that each $S_{i}$ contains exactly one atom of $G$.

Governed by the above observations, by substituting $A=(\epsilon/\text{Diam}(\Theta))^{r}$, we obtain that

[TABLE]

By means of Dirichlet probability model assumption on $\Delta_{D-1}$, we have the following evaluations with $T$

[TABLE]

where $\mathcal{U}:=\Delta_{D-1}\cap|G(S_{i})-G_{*}(S_{i})|\leq(\epsilon/\text{Diam}(\Theta))^{r}/D$. Here, the second inequality in the above display is due to the fact that $(1-\sum_{i=1}^{D-1}G(S_{i}))^{(\gamma/D)-1}>1$ as $\gamma<D$. Invoking the basic inequality $\alpha\Gamma(\alpha)<1$ for $0<\alpha<1$, we reach the conclusion of the lemma.

6.4 Proof of Proposition 4.1

We denote a sphere of radius $R$ as $S_{R}:=\{x\in\mathbb{R}^{d}:\|x\|_{2}\leq R\}$ for any $R>0$. Direct computations lead to

[TABLE]

where the last inequality is an application of Holder’s inequality. Now, a direct evaluation yields that

[TABLE]

The last inequality is due to the fact that $(x-\theta)^{\top}\Sigma^{-1}(x-\theta)\geq\|x-\theta\|_{2}^{2}/\lambda_{\text{max}}$ for all $x\in\mathbb{R}^{d}$ and $\theta\in\Theta$.

We now assume that $d>2$ as the $d\leq 2$ case can be treated similarly. Since $\Theta\subset\mathbb{R}^{d}$ is bounded, we can find $r>0$ such that $\|\theta\|_{2}<r$ for all $\theta\in\Theta$. Now, given $R>r$, for any fixed value of $x\in S_{R}^{c}$, we can check that $\inf_{\theta}\|x-\theta\|_{2}^{2}\geq(\|x\|_{2}-r)^{2}$. Therefore equation (30) leads to

[TABLE]

Invoking spherical coordinates by substituting $z=\|x\|_{2}$, we get

[TABLE]

We denote $g_{R}(d):=\int_{z>R}{z^{d}}\exp(-(z-r)^{2})dz$. By integrating by parts with some basic algebraic manipulation, we find that

[TABLE]

Observe that $(R^{d-2}/2)\exp(-(R-r)^{2})\gtrsim(R^{s}/2)\exp(-(R-r)^{2})$ for all $s\leq d-2$. Also, $\int_{x}^{\infty}\exp(-t^{2}/2)\mathrm{d}t\leq\frac{1}{x}\exp(-x^{2}/2)\lesssim x^{d-2}\exp(-x^{2}/2)$ and $\int_{x}^{\infty}t\exp(-t^{2}/2)\mathrm{d}t\lesssim\exp(-x^{2}/2)\lesssim x^{d-2}\exp(-x^{2}/2)$ follows using standard arguments for gaussian tailbounds. Here we use the condition $d>2$. For $d\leq 2$, the tail probability can be directly bounded using standard gaussian tailbounds.

We can expand $g_{R}(s)$ recursively using Equation 31 for all $s\leq d-2$ as well. Now, equipped with equation (31), and following the discussion in the previous paragraph, we can write

[TABLE]

Now, we demonstrate that $\left\lVert\frac{p_{G_{*}}(\cdot)}{p_{G_{0},f_{0}}(\cdot)}\mathbbm{1}_{S_{R}}\right\rVert_{\infty}$ is bounded above by $c_{2}\exp(\lambda_{\min}^{-1}R^{2})$ for some positive constant $c_{2}$ depending only on $C_{1},G_{0}$ and $\lambda_{\min}$. Recall that $G_{0}=\sum_{i=1}^{k_{0}}p_{i}^{0}\delta_{\theta^{0}_{i}}$. Here $k_{0}$ can be allowed to be $\infty$. The analysis follows through similar to the finite $k_{0}$ case.

The conditions on $p_{G_{0},f_{0}}$ imply that

[TABLE]

Then, we have the following inequalities

[TABLE]

The bounds apply uniformly for all $R>r$. Therefore, when $R\geq 4r$, we can bound equation (29) according to the bounds in (32) and (33) as follows:

[TABLE]

where $\overline{V}(p_{G},p_{G^{\prime}}):=\int_{\mathbb{R}^{d}}|p_{G}(x)-p_{G^{\prime}}(x)|\frac{p_{G_{0},f_{0}}(x)}{p_{G_{*}}(x)}\mu(\mathrm{d}x)$ is the weighted variational distance. Now consider $R>0$ satisfying $R^{d-2}\leq\exp(-\lambda_{\max}^{-1}R^{2}/8)$. If $\overline{V}(p_{G},p_{G^{\prime}})=\exp(-(\lambda_{\min}^{-1}+C_{0}+\lambda_{\max}^{-1}/8)R^{2})$, we obtain that

[TABLE]

Note that, $\overline{V}(p_{G},p_{G^{\prime}})\lesssim\overline{h}(p_{G},p_{G^{\prime}})$ by standard application of Holder’s inequality. Also, since the kernel for location Gaussian mixtures is supersmooth [11], it follows from Theorem 2 in [37] that V(p_{G},p_{G^{\prime}})\gtrsim\exp\biggr{(}-1/W_{2}^{2}(G,G^{\prime})\biggr{)}. The proof of the proposition now follows from this fact, and the inequality connecting weighted Hellinger and variational distances.

6.5 Proof of Proposition 4.2

We denote a sphere of radius $R$ as $S_{R}:=\{x\in\mathbb{R}^{d}:\|x\|_{2}\leq R\}$ for any $R>0$. Assume $\max_{\theta\in\Theta}\|\theta\|_{2}\leq r$ and also $\sup_{i\leq k_{0}}\|\theta_{i}^{0}\|_{2}\leq r$. We consider $R>2r$ large enough such that, $\|\theta\|_{2}\leq r$ and $\|x\|_{2}\geq R$ implies

[TABLE]

The above inequalities can always be achieved for $R$ large enough because of the asymptotic formulation of multivariate Laplace distributions.

Following equation (29) in the proof of Theorem 4.1, we will prove the proposition by providing upper bounds for $\left\lVert\frac{p_{G_{*}}(\cdot)}{p_{G_{0},f_{0}}(\cdot)}\mathbbm{1}_{S_{R}}\right\rVert_{\infty}$ and $\int_{S_{R}^{\mathsf{c}}}|p_{G}(x)-p_{G^{\prime}}(x)|\mathrm{d}\mu(x)$. Because the Laplace density is bounded, $\left\lVert p_{G_{*}}(\cdot)\mathbbm{1}_{S_{R}}\right\rVert_{\infty}$ is bounded by a constant. Similar to the proof of Proposition 4.1, we have,

[TABLE]

Now, in order to minimize $\int_{S_{R}^{\mathsf{c}}}|p_{G}(x)-p_{G^{\prime}}(x)|\mathrm{d}\mu(x)$, observe that

[TABLE]

Substituting $z=\|x\|_{2}$ in above equation, we get

[TABLE]

Denote $g_{R}(s):=\int_{S_{R}^{\mathsf{c}}}{z^{s}}\exp(-z)dz$. Then, we find that

[TABLE]

Invoking integration by parts with the above equality for $s=\frac{d-1}{2}$ leads to the following inequality

[TABLE]

Since the above bounds apply for all $R$ large enough, following the approach with equation (29) in the proof of Theorem 4.1, we can write

[TABLE]

Recall that $\overline{V}(p_{G},p_{G^{\prime}}):=\int_{\mathbb{R}^{d}}|p_{G}(x)-p_{G^{\prime}}(x)|\frac{p_{G_{0}}(x)}{p_{G_{*}}(x)}\mu(\mathrm{d}x)$ is the weighted variational distance. By setting $\overline{V}(p_{G},p_{G^{\prime}})=\exp\left(-\left[\sqrt{\frac{2}{\lambda\lambda_{\min}}}+\sqrt{\frac{2}{\lambda\lambda_{\max}}}+C_{0}\right]R^{\alpha}\right)$, as $\alpha\geq 1$, we see that

[TABLE]

where \tau(\alpha)=\sqrt{\frac{2}{\lambda\lambda_{\max}}}\bigg{/}\left[\sqrt{\frac{2}{\lambda\lambda_{\max}}}+\sqrt{\frac{2}{\lambda\lambda_{\min}}}+C_{0}\right]^{1/\alpha}. Now, the location family of multivariate Laplace distributions pertains to the ordinary smooth likelihood families. Therefore, from part (1) of Theorem 2 in [37], it follows that for any $m<4/(4+5d)$, $W_{2}^{2}(G,G^{\prime})\leq V(p_{G},p_{G^{\prime}})^{m}$. We note in passing that improved rates for other choices of $W_{r}$ may be possible by utilizing techniques similar to [14]. Thus, by means of the inequality $\overline{V}(p_{G},p_{G^{\prime}})\lesssim\overline{h}(p_{G},p_{G^{\prime}})$, the following inequality holds

[TABLE]

The result now follows by taking logarithms of both sides.

Acknowledgements

This research is supported in part by grants NSF CAREER DMS-1351362, NSF CNS-1409303, a research gift from Adobe Research and a Margaret and Herman Sokol Faculty Award.

7 Appendix A: Weighted Hellinger and Wasserstein distance

In this appendix, we will establish several useful bounds between weighted Hellinger distance and Wasserstein metric that are employed in the proofs for misspecified settings of Section 4. See the formal setup of $G_{0},f_{0}$ and $G_{*}$ in the beginning of that section.

First, we start with the following lemma regarding an upper bound of weighted Hellinger distance in terms of Wasserstein metric when the kernel $f$ satisfies first order integral Lipschitz condition.

Lemma 7.1**.**

Assume that the kernel $f$ is integral Lipschitz up to the first order. Then, for any mixing measure $G_{1}$ and $G_{2}$ in $\mathcal{P}(\Theta)$, there exists a positive constant $\overline{C}(\Theta)$ depending only on $\Theta$ such that

[TABLE]

Proof.

Denote the weighted total variation distance as follows

[TABLE]

for any $G_{1},G_{2}\in\mathcal{P}(\Theta)$. It is clear that $\overline{h}^{2}(p_{G_{1}},p_{G_{2}})\leq\overline{V}(p_{G_{1}},p_{G_{2}})$ for any $G_{1},G_{2}\in\mathcal{P}(\Theta)$.

For any coupling $\boldsymbol{q}$ of the weight vectors of $G_{1}=\sum\limits_{i=1}^{k_{1}}{p_{i,1}\delta_{\theta_{i,1}}}$ and $G_{2}=\sum\limits_{i=1}^{k_{2}}{p_{i,2}\delta_{\theta_{i,2}}}$, we can check via triangle inequality that

[TABLE]

where the existence of positive constant $\overline{C}(\Theta)$ in the second inequality is due to the first order integral Lipschitz property of $f$. The above result implies that

[TABLE]

for any $G_{1},G_{2}\in\mathcal{P}(\Theta)$. We achieve the conclusion of the lemma. ∎

7.1 Proof of Lemma 4.2

The proof is a straightforward application of Lemma 4.1. In fact, from that lemma, we have

[TABLE]

where the first inequality is due to Cauchy inequality. The above inequality holds only when $p_{G_{1,*}}(x)=p_{G_{2,*}}(x)$ for almost all $x\in\mathcal{X}$, which concludes our lemma.

7.2 Proof of Proposition 4.3

Denote the weighted total variation distance as follows

[TABLE]

for any $G_{1},G_{2}\in\mathcal{G}$. Then, by means of Holder’s inequality, we can verify that

[TABLE]

where the last inequality is due to Lemma 4.1. Therefore, to obtain the conclusion of the proposition, it is sufficient to demonstrate that

[TABLE]

where $\overline{k}>k_{*}$. Firstly, we will show that

[TABLE]

Assume that the above inequality does not hold. It implies that there exists a sequence of $G_{n}\in\mathcal{O}_{\overline{k}}(\Theta)$ such that $\overline{V}(p_{G_{n}},p_{G_{*}})/W_{2}^{2}(G_{n},G_{*})\to 0$ as $n\to\infty$. By means of Fatou’s lemma, we have

[TABLE]

Hence, for almost surely $x\in\mathcal{X}$, we obtain that

[TABLE]

The above equality is equivalent to

[TABLE]

for almost surely $x\in\mathcal{X}$. However, using the same argument as that of Theorem 3.2 in [22], the above equality cannot hold due to the second order identifiability of $f$, which is a contradiction. Therefore, we can find positive constant $\epsilon_{0}>0$ such that as long as $W_{2}(G,G_{*})\leq\epsilon_{0}$, we achieve that $\overline{V}(p_{G},p_{G_{*}})\gtrsim W_{2}^{2}(G,G_{*})$. As a consequence, to obtain the conclusion of (34), we only need to verify that

[TABLE]

Assume that the above result does not hold. It implies that we can find a sequence of $G_{n}\in\mathcal{O}_{\overline{k}}$ such that $W_{2}(G_{n},G_{*})>\epsilon_{0}$ and $\overline{V}(p_{G_{n}},p_{G_{*}})/W_{2}^{2}(G_{n},G_{*})\to 0$ as $n\to\infty$. Since $\Theta$ is a bounded subset of $\mathbb{R}^{d}$, we can find a subsequence of $G_{n}$ such that $W_{1}(G_{n},\overline{G})\to 0$ for some $\overline{G}\in\mathcal{O}_{\overline{k}}$ such that $W_{2}(\overline{G},G_{*})\geq\epsilon_{0}$. From our hypothesis, we will have that $\overline{V}(p_{G_{n}},p_{G_{*}})\to 0$. However, by virtue of Fatou’s lemma, we obtain that

[TABLE]

The above equation leads to $p_{\overline{G}}(x)=p_{G_{*}}(x)$ for almost surely $x\in\mathcal{X}$. Due to the identifiability of $f$, the previous equation leads to $\overline{G}\equiv G_{*}$, which is a contradiction to the assumption that $W_{2}(\overline{G},G_{*})\geq\epsilon_{0}$. We obtain the conclusion of the proposition.

8 Appendix B: Posterior contraction under misspecification

This appendix is devoted to the description of a general method for establishing posterior convergence rates of mixing measures under misspecified settings, extending the methods of [27] and [37]. Once the general method is fully developed we shall be ready to complete the proofs of the main theorems of Section 4, which are given in Section 9. Recall the weighted Hellinger distance defined in (4.1), which leads to the following definition.

Definition 8.1**.**

For any set $\mathcal{S}\subset\overline{\mathcal{G}}$, define a real-valued function $\overline{\Psi}_{\mathcal{S}}:\mathbb{R}\to\mathbb{R}^{+}$ as follows

[TABLE]

for any $r\in\mathbb{R}$.

A key ingredient to establishing the posterior contraction bounds is through the existence of tests for subsets of parameters of interest. In the model misspecification setting, it is no longer appropriate to test any mixing measure $G$ against true measure $G_{0}$. Instead, following [27], it is appropriate to test any mixing measure $G$ against $G_{*}$, which is ultimately achieved by testing $\dfrac{p_{G_{0},f_{0}}}{p_{G_{*}}}p_{G}$ against $p_{G_{0},f_{0}}$. This insight leads us to the following crucial result regarding the existence of test for discriminating $G_{*}$ against a closed Wasserstein metric ball centered at $G_{1}$ for any fixed pair of mixing measures $(G_{*},G_{1})$.

Lemma 8.1**.**

Consider $\mathcal{S}\in\overline{\mathcal{G}}$ such that $G_{*}\in\mathcal{S}$. Given $G_{1}\in\mathcal{S}$ such that $W_{2}(G_{1},G_{*})\geq r$ for some $r>0$. Assume that either one of the following two sets of conditions holds:

• (1)

$\mathcal{S}$* is a convex set, in which case, let $\overline{M}(\mathcal{S},G_{1},r)=1$.*

• (2)

$\mathcal{S}$* is a nonconvex set. In addition, $f$ has first order integral Lipschitz property. In this case, we define that*

[TABLE]

Then, there exists tests $\phi_{n}$ such that

[TABLE]

By means of the existence of tests in Lemma 8.1, we have the following result regarding testing $G_{*}$ versus a complement of a closed Wasserstein ball.

Lemma 8.2**.**

Assume that all the conditions in Lemma 8.1 hold. Let $D(\epsilon)$ be a non-decreasing function such that , for some $\epsilon_{n}\geq 0$ and every $\epsilon>\epsilon_{n}$,

[TABLE]

Then, for every $\epsilon>\epsilon_{n}$ there exist tests $\phi_{n}$ (depending on $\epsilon>0$) such that, for every $J\in\mathbb{N}$,

[TABLE]

For any $\epsilon>0,M>0$, we define a generalized Kullback-Leibler neighborhood of $G_{*}$ by

[TABLE]

Invoking the results in Lemma 8.1 and Lemma 8.2, we have the following theorem establishing posterior contraction rate for $G_{*}$. This theorem generalizes Theorem 3 in [37] to the misspecified setting.

Theorem 8.1**.**

Suppose that for a sequence of $\left\{\epsilon_{n}\right\}_{n\geq 1}$ that tends to a constant (or [math]) such that $n\epsilon_{n}^{2}\to\infty$, and constants $C,M>0$, and convex sets $\mathcal{G}_{n}\subset\overline{\mathcal{G}}$, we have

[TABLE]

Additionally, $M_{n}$ is a sequence such that

[TABLE]

Then, $\Pi(G\in\overline{\mathcal{G}}:\ W_{2}(G,G_{*})\geq M_{n}\epsilon_{n}|X_{1},\ldots,X_{n})\to 0$ in $P_{G_{0},f_{0}}$-probability.

The above theorem is particularly useful for establishing the convergence rate of $G_{*}\in\mathcal{P}(\Theta)$ for which the suitable sieves $\mathcal{G}_{n}$ of mixture densities are convex classes of functions. In the situation where $\mathcal{G}_{n}$ are non-convex, we need the following result, which is the generalization of Theorem 4 in [37].

Theorem 8.2**.**

Assume that $f$ admits the first order integral Lipschitz property. Additionally, there is a sequence $\epsilon_{n}$ with $\epsilon_{n}\to 0$ such that $n\log(1/\epsilon_{n})^{-2}$ is bounded away from 0, a sequence $M_{n}$, a constant $M>0$, and a sequence of sets $\mathcal{G}_{n}\subset\overline{\mathcal{G}}$ such that the following conditions hold

[TABLE]

Then, we have that $\Pi(G\in\overline{\mathcal{G}}:W_{2}(G,G_{*})\geq M_{n}\epsilon_{n}|X_{1},\ldots,X_{n})\to 0$ in $P_{G_{0},f_{0}}$- probability.

9 Appendix C

We are now ready to complete the proof of the main posterior contraction theorems stated in Section 4.

9.1 Proof of Theorem 4.1

Note that the MFM prior places full mass on discrete measure with finite support, it is enough to show \Pi\biggr{(}G\in\mathcal{G}(\Theta):W_{2}(G,G_{*})\gtrsim\left(\frac{\log\log n}{\log n}\right)^{1/2}\biggr{|}X_{1},\ldots,X_{n}\biggr{)}\to 0.

The proof of this result is a straightforward application of Proposition 4.1, Lemma 4.3, and Theorem 8.1; therefore, we will only provide a sketch of this proof. Similar to the proof of Theorem 3.1 (for the well-specified setting), we proceed by constructing a sequence $\epsilon_{n}$ and sieves $\mathcal{G}_{n}$ that satisfy all the conditions specified in Theorem 8.1.

Step 1:

First, we choose $\epsilon_{n}$ to satisfy condition (42) in Theorem 8.1. To that effect, we proceed by making use of the results from Lemma 8.1 in [27]. In particular, from Lemma 8.1 of [27], as long as $P$ is a probability measure and $Q$ is a finite measure (with densities $p$ and $q$ respectively, with respect to Lebesgue measure on $\mathbb{R}^{d}$) such that $h(p,q)\leq\epsilon$ and $\int p^{2}/q\leq M$, we obtain that

[TABLE]

where the constants in these bounds are universal. For the purpose of our proof, we will choose $p=p_{G_{0},f_{0}}$ and $q=p_{G}p_{G_{0},f_{0}}/p_{G_{*}}$.

Since the Gaussian kernel satisfies the integral Lipschitz property up to the first order, by invoking the result of Lemma 7.1, we have

[TABLE]

Additionally, for the Gaussian kernel $f$ given by (12), we find that

[TABLE]

for any $G\in\overline{\mathcal{G}}(\Theta)$. For Gaussian location mixtures as long as there exists $\epsilon_{0}>0$ such that $W_{2}(G,G_{*})\leq\epsilon_{0}$, we also can check that $\int p_{G_{0},f_{0}}(x)p_{G_{*}}(x)/p_{G}(x)\mathrm{d}\mu(x)\leq M^{*}(\epsilon_{0})$ for some positive constant $M^{*}(\epsilon_{0})$ depending only on $\epsilon_{0},G_{0}$, and $\Theta$. Therefore, as long as $W_{2}(G,G_{*})\leq\epsilon\leq\epsilon_{0}$, we have for mixing measure $G$,

[TABLE]

where $M:=M^{*}(\epsilon_{0})$. The constants in the bounds are all universal.

Governed by this result, we can write

[TABLE]

for any sequence $\epsilon_{n}\leq\epsilon_{0}$. Now, the packing number $D=D(\epsilon_{n})$ (with packing radius $\epsilon_{n}$) in Lemma 4.3 satisfies $D(\epsilon_{n})\asymp\left(\frac{\text{Diam}(\Theta)}{\epsilon_{n}}\right)^{d}$. Following the result in Lemma 4.3, we have with $r=2$ that

[TABLE]

With $\epsilon_{n}\asymp(\frac{\log n}{n})^{1/(d+2)}$, one can check that condition (42) and  (40) hold.

Step 2:

Note that condition (41) holds automatically since we take $\mathcal{G}_{n}=\overline{\mathcal{G}}$, while condition (40) follows from Lemma 4 in [37].

Step 3:

Next we will show condition (43) and condition (44) for some appropriate choice of $M_{n}$ for the $\epsilon_{n}$ considered in Step 1.

Following proposition 4.1 we know that \overline{\Psi}_{\mathcal{G}_{n}}(r)\gtrsim\exp\biggr{(}-(1+(2\lambda_{max}/\lambda_{min}))/r^{2}\biggr{)}. Using this fact, we can check to see that $M_{n}$ such that $M_{n}\epsilon_{n}\approx\left(\frac{\log\log(n)}{\log n}\right)^{1/2}$ works, with $\epsilon_{n}\approx(\frac{\log n}{n})^{1/(d+2)}$.

9.2 Proof of Lemma 8.2

Consider a $t\epsilon/2$ maximal-packing of the set $B_{W_{1}}(G_{*},2t\epsilon)\backslash B_{W_{1}}(G_{*},t\epsilon)$. Let $S_{t}$ be the corresponding set of $D(t\epsilon/2,\mathcal{S}\cap B_{W_{1}}(G_{*},2t\epsilon)\backslash B_{W_{1}}(G_{*},t\epsilon)$ points obtained there in. Then as in Lemma 8.1 corresponding to each point $G_{1}$ in $S_{t}$, there exist $\omega_{n,t}$ which satisfies (35) and (36). Then taking $\phi_{n}$ as the supremum over all these tests $\omega_{n,t}$ over all points in $S_{t}$ , all $t\geq J$ we see that by the union bound,

[TABLE]

The last inequality follows from the fact that $\overline{\Psi}_{\mathcal{S}}(\cdot)$ is an increasing function in its argument.

9.3 Proof of Theorem 8.2

The following lemma is analogous to Lemma 7.1 in [27] and Lemma 8.1 in [16] and can be similarly proved.

Lemma 9.1**.**

For every $M,\epsilon>0$, $C>0$, and probability measure $\Pi$ on $G$, we obtain that

[TABLE]

Equipped with this lemma we can now prove the theorem as follows. Denote $A_{n}$ the event such that

[TABLE]

The above result indicates that $P_{G_{0},f_{0}}\mathbbm{1}_{A_{n}}\leq(C^{2}n)^{-1}\log(M/\epsilon)^{2}$ for any $\epsilon>0$ and $C>0$.

For any sequence $\epsilon_{n}$, we denote $\mathcal{U}_{n}=\left\{G\in\overline{\mathcal{G}}:W_{2}(G,G_{*})\geq M_{n}\epsilon_{n}\right\}$ and $S_{n,j}=\left\{G\in\mathcal{G}_{n}:\right.\\ \left.W_{2}(G,G_{*})\in[j\epsilon_{n},(j+1)\epsilon_{n})\right\}$ for any $j\geq 1$. From the result of Lemma 8.1 and condition (53), there exists a test $\phi_{n}$ such that inequality  (37) and (38) hold when $D(\epsilon_{n})=\exp(n\epsilon_{n}^{2})$. Now, we have

[TABLE]

According to Lemma 8.2, we have $P_{G_{0},f_{0}}\phi_{n}\leq\exp(n\epsilon_{n}^{2})\sum\limits_{j\geq M_{n}}{\exp\left(-n\overline{\Psi}_{\mathcal{G}_{n}}(j\epsilon_{n})/8\right)}\to 0$, which is due to condition (48). Additionally, from the formation of $A_{n}$, we also obtain that $P_{G_{0},f_{0}}\mathbbm{1}_{A_{n}}\leq(C^{2}n)^{-1}\log(M/\epsilon)^{2}$. If $n\log(1/\epsilon)^{-2}\to\infty$, it is clear that $P_{G_{0},f_{0}}\mathbbm{1}_{A_{n}}\to 0$ for any $C\geq 1$. If $n\log(1/\epsilon)^{-2}$ does not tend to $\infty$ but is bounded away from 0, then we can choose $C>0$ large enough such that $P_{G_{0}}\mathbbm{1}_{A_{n}}$ is sufficiently close to 0. Therefore, the first two terms in (53) can always be made to vanish to 0. To achieve the conclusion of the theorem, it is sufficient to demonstrate that the third term in (52) goes to 0. In fact, we have the following equation

[TABLE]

From the formulation of $A_{n}$, we have

[TABLE]

By means of Fubini’s theorem, we obtain that

[TABLE]

where the last inequality is due to inequality (38) and condition (53). Furthermore, by means of Fubini’s theorem

[TABLE]

where the second inequality in the above result is due to Lemma 4.1. By combining the results of (53), (54), and (55), we obtain

[TABLE]

where the last inequality is due to condition (54) and (55). If $n\log(1/\epsilon_{n})^{-2}$ is bounded away from 0 by choosing $C\geq 1$, the right hand side term of the above display will go to 0 due to condition (48). Therefore, we have $P_{G_{0},f_{0}}(1-\phi_{n})\mathbbm{1}_{A_{n}^{c}}\Pi(G\in\mathcal{U}_{n}|X_{1},\ldots,X_{n})\to 0$ as $n\to\infty$.

As a consequence, $P_{G_{0},f_{0}}\Pi(G\in\mathcal{U}_{n}|X_{1},\ldots,X_{n})\to 0$. We achieve the conclusion of the theorem.

9.4 Proof of Lemma 8.1

For the setting (1) when $\mathcal{S}$ is convex, since $B_{W_{2}}(G_{1},r/2)$ is a convex set, we also have $\mathcal{S}\cap B_{W_{2}}(G_{1},r/2)$ is a convex set. By means of the result of Theorem 6.1 in [27], there exist tests $\phi_{n}$ such that

[TABLE]

Due to inequality $(1-x)^{n}\leq\exp(-nx)$ for all $0<x<1$ and $n\geq 1$, the above inequalities become

[TABLE]

Now, since $W_{2}(G_{1},G_{*})=r$ and $W_{2}(G,G_{1})\leq r/2$ as long as $G\in\mathcal{S}\cap B_{W_{2}}(G_{1},r/2)$, it implies that $W_{2}(G_{1},G_{*})\geq r/2$. Therefore, according to Definition 8.1, we will obtain that

[TABLE]

With the above inequality, we reach the conclusion of part (1).

Regarding part (2), we consider a maximal $c_{0}r$-packing of $\mathcal{S}\cap B_{W_{2}}(G_{1},r/2)$ under $W_{2}$ metric. It gives us a set of $\overline{M}=\overline{M}(\mathcal{S},G_{1},r)=D(c_{0}r,\mathcal{S}\cap B_{W_{2}}(G_{1},r/2),W_{2})$ points $\widetilde{G}_{1},\ldots,\widetilde{G}_{\overline{M}}$ in $\mathcal{S}\cap B_{W_{2}}(G_{1},r/2)$.

Now, for any $G\in\mathcal{S}\cap B_{W_{2}}(G_{1},r/2)$, we can find $t\in\left\{1,\ldots,\overline{M}\right\}$ such that $W_{1}(G,\widetilde{G}_{t})\leq c_{0}r$. Due to the triangle inequality, we achieve that

[TABLE]

where the second inequality is due to Definition 8.1 and Lemma 7.1. By choosing the positive number $c_{0}=\overline{\Psi}_{\mathcal{S}}(r)/(4\overline{C}(\Theta)r)$, we obtain

[TABLE]

It eventually leads to $\overline{h}(p_{G},p_{G_{*}})\geq\overline{h}(p_{G_{*}},p_{\widetilde{G}_{t}})/2$. According to the result of Theorem 6.1 in [27] and inequality $(1-x)^{n}\leq\exp(-nx)$ for all $0<x<1$ and $n\geq 1$, by denoting

[TABLE]

there exists test $\psi_{n}^{(t)}$ such that

[TABLE]

Since $\overline{h}^{2}(p_{G},p_{G_{*}})\geq\overline{h}(p_{G_{*}},p_{\widetilde{G}_{t}})/2\geq\overline{\Psi}_{\mathcal{S}}(r)/2$ for all $G\in A_{t}$, the above inequalities can be rewritten as

[TABLE]

By choosing $\phi_{n}=\max\limits_{1\leq t\leq\overline{M}}\psi_{n}^{(t)}$, we quickly achieve that

[TABLE]

As a consequence, we obtain the conclusion of the lemma.

9.5 Proof of Theorem 4.3

Due to the assumption on prior $p_{K}$, it is sufficient to demonstrate that

[TABLE]

in $P_{G_{0},f_{0}}$- probability. We divide our proof for the above result into the following steps

Step 1:

To obtain the bound for $\Pi(B_{K}^{*}(\epsilon_{n},G_{*},P_{G_{0},f_{0}},M))$, we use Lemma 8.1 from [27] to obtain a bound for weighted KL divergence and squared weighted KL divergence. Similar to the proof of Theorem 4.1, with the choice of $p=P_{G_{0},f_{0}}$ and of finite measure $q=p_{G}p_{G_{0},f_{0}}/p_{G_{*}}$, as long as $G\in\mathcal{O}_{\overline{k}}$ such that $\overline{h}(p_{G},p_{G_{*}})\leq\epsilon$ and $\int p_{G_{0},f_{0}}p_{G_{*}}/p_{G}\leq M$ then we obtain that

[TABLE]

For the purpose of this proof we use $M=M^{*}(\epsilon_{0})$, where $M^{*}(\epsilon_{0})$ is as in condition (M.2) in Section 4.3. Now, according to Lemma 7.1, as $f$ admits integral Lipschitz property up to the first order, we obtain that $\overline{h}^{2}(p_{G},p_{G_{*}})\leq\overline{C}W_{1}(G,G_{*})$ for any $G\in\mathcal{O}_{\overline{k}}$ where $\overline{C}$ is a positive constant depending only on $\Theta$. Now, from the discussion in the above paragraph we have

[TABLE]

where the last inequality can be obtained similar to equation (19) based on the assumption $\gamma<\overline{k}$. Now, we note that $\overline{\Psi}_{\mathcal{G}_{n}(r)}\gtrsim r^{4}$ , since $f$ is assumed to be second order identifiable and to satisfy the integral Lipschitz property of second order. Then, by choosing $\epsilon_{n}=n^{-1}$ and $M_{n}=\overline{A}\left(\frac{\log(n)}{n^{3}}\right)^{1/4}$ for some sufficiently large $\overline{A}$, we can see that condition (47) is satisfied.

Step 2:

We choose the sieves $\mathcal{G}_{n}=\mathcal{O}_{\overline{k}}$ for all $n\geq 1$. With these choices, it is clear that $\Pi(\overline{\mathcal{G}}\backslash\mathcal{G}_{n})=0$. Therefore, condition (46) is satisfied.

Step 3:

For condition (48) to be satisfied,

[TABLE]