Objective Bayesian Analysis for Change Point Problems

TL;DR

This paper introduces a loss-based Bayesian method for change point analysis, addressing both prior specification with known change points and estimating their number as a model selection problem, validated on simulated and real data.

Contribution

It presents a novel loss-based Bayesian framework for change point detection and model selection, expanding the methodological toolkit in this area.

Findings

Effective on simulated data

Successful application to real data

Improves change point estimation accuracy

Abstract

In this paper we present a loss-based approach to change point analysis. In particular, we look at the problem from two perspectives. The first focuses on the definition of a prior when the number of change points is known a priori. The second contribution aims to estimate the number of change points by using a loss-based approach recently introduced in the literature. The latter considers change point estimation as a model selection exercise. We show the performance of the proposed approach on simulated data and real data sets.

Figures

Tables

Table 1. Table 1: Model prior, Bayes factor and model posterior probabilities for the change point analysis in Scenario 1. We considered samples from, respectively, model M 0 subscript 𝑀 0 M_{0} and model M 1 subscript 𝑀 1 M_{1} .

•	True model
•	$M_0$	$M_1$
$\mathrm{Pr}\left(M_0\right)$	0.47	0.47
$\mathrm{Pr}\left(M_1\right)$	0.53	0.53
$B_{01}$	12.39	0.08
$B_{10}$	0.08	12.80
$\mathrm{Pr}\left(M_0|{𝐱}^{\left(n\right)}\right)$	0.92	0.06
$\mathrm{Pr}\left(M_1|{𝐱}^{\left(n\right)}\right)$	0.08	0.94

Table 2. Table 2: Model prior, Bayes factor and model posterior probabilities for the change point analysis in Scenario 2. We considered samples from, respectively, model M 0 subscript 𝑀 0 M_{0} , model M 1 subscript 𝑀 1 M_{1} and model M 2 subscript 𝑀 2 M_{2} .

•	True model
•	$M_0$	$M_1$	$M_2$
$\mathrm{Pr}\left(M_0\right)$	0.27	0.27	0.27
$\mathrm{Pr}\left(M_1\right)$	0.39	0.39	0.39
$\mathrm{Pr}\left(M_2\right)$	0.34	0.34	0.34
$B_{01}$	36.55	$3.24\times {10}^{-4}$	$4.65\times {10}^{-40}$
$B_{02}$	$1.84\times {10}^3$	0.02	$1.27\times {10}^{-45}$
$B_{12}$	50.44	55	$2.72\times {10}^{-6}$
$\mathrm{Pr}\left(M_0|{𝐱}^{\left(n\right)}\right)$	0.96	0.00	0.00
$\mathrm{Pr}\left(M_1|{𝐱}^{\left(n\right)}\right)$	0.04	0.98	0.00
$\mathrm{Pr}\left(M_2|{𝐱}^{\left(n\right)}\right)$	0.00	0.02	1.00

Table 3. Table 3: Average model posterior probabilities, variance and frequency of true model for the Scenario 3 simulation exercise.

•	Mean posterior	Variance posterior	\pbox15cm Freq. true model
$\mathrm{Pr}\left(M_0|{𝐱}^{\left(n\right)}\right)$	0.01	$3.84\times {10}^{-4}$	0/60
$\mathrm{Pr}\left(M_1|{𝐱}^{\left(n\right)}\right)$	0.24	0.0160	0/60
$\mathrm{Pr}\left(M_2|{𝐱}^{\left(n\right)}\right)$	0.75	0.0190	60/60

Table 4. Table 4: Average model posterior probabilities, variance and frequency of true model for the Scenario 4 simulation exercise with n = 500 𝑛 500 n=500 and the loss-based prior.

•	Mean posterior	Variance posterior	\pbox15cm Freq. true model
$\mathrm{Pr}\left(M_0|{𝐱}^{\left(n\right)}\right)$	$9.88\times {10}^{-4}$​	$2.60\times {10}^{-5}$	0/100
$\mathrm{Pr}\left(M_1|{𝐱}^{\left(n\right)}\right)$	0.63	0.0749	70/100
$\mathrm{Pr}\left(M_2|{𝐱}^{\left(n\right)}\right)$	0.37	0.0745	30/100

Table 5. Table 5: Average model posterior probabilities, variance and frequency of true model for the Scenario 4 simulation exercise with n = 1500 𝑛 1500 n=1500 and the loss-based prior.

•	Mean posterior	Variance posterior	\pbox15cm Freq. true model
$\mathrm{Pr}\left(M_0|{𝐱}^{\left(n\right)}\right)$	$1.33\times {10}^{-13}$	$1.76\times {10}^{-24}$	0/100
$\mathrm{Pr}\left(M_1|{𝐱}^{\left(n\right)}\right)$	0.08	0.0200	4/100
$\mathrm{Pr}\left(M_2|{𝐱}^{\left(n\right)}\right)$	0.92	0.0200	96/100

Table 6. Table 6: Average model posterior probabilities, variance and frequency of true model for the Scenario 4 simulation exercise with n = 500 𝑛 500 n=500 and the uniform prior.

•	Mean posterior	Variance posterior	\pbox15cm Freq. true model
$\mathrm{Pr}\left(M_0|{𝐱}^{\left(n\right)}\right)$	$16\times {10}^{-4}$	$7.15\times {10}^{-5}$	0/100
$\mathrm{Pr}\left(M_1|{𝐱}^{\left(n\right)}\right)$	0.82	0.0447	91/100
$\mathrm{Pr}\left(M_2|{𝐱}^{\left(n\right)}\right)$	0.18	0.0443	9/100

Table 7. Table 7: Average model posterior probabilities, variance and frequency of true model for the Scenario 4 simulation exercise with n = 1500 𝑛 1500 n=1500 and the uniform prior.

•	Mean posterior	Variance posterior	\pbox15cm Freq. true model
$\mathrm{Pr}\left(M_0|{𝐱}^{\left(n\right)}\right)$	$8.64\times {10}^{-12}$	$7.45\times {10}^{-21}$	0/100
$\mathrm{Pr}\left(M_1|{𝐱}^{\left(n\right)}\right)$	0.501	0.1356	49/100
$\mathrm{Pr}\left(M_2|{𝐱}^{\left(n\right)}\right)$	0.499	0.1356	51/100

Table 8. Table 8: Hellinger distances between all the pairs formed from a Weibull( λ , κ 𝜆 𝜅 \lambda,\kappa ), Log-normal( μ , τ 𝜇 𝜏 \mu,\tau ) and Gamma( α , β 𝛼 𝛽 \alpha,\beta ). The six hyperparameters are such that the distributions have the same mean=5 and same variance=2.5.

•	Hellinger distances
•	Weibull($\lambda ,\kappa$)	Log-normal($\mu ,\tau$)	Gamma($\alpha ,\beta$)
Weibull($\lambda ,\kappa$)		0.1411996	0.09718282
Log-normal($\mu ,\tau$)			0.04899711

Table 9. Table 9: Model prior, Bayes factor and model posterior probabilities for the S&P500 change point analysis.

$\mathrm{Pr}\left(M_0\right)$	0.36
$\mathrm{Pr}\left(M_1\right)$	0.32
$\mathrm{Pr}\left(M_2\right)$	0.32
$B_{01}$	$7.72\times {10}^{18}$
$B_{02}$	$3.30\times {10}^{-3}$
$B_{12}$	$4.28\times {10}^{-22}$
$\mathrm{Pr}\left(M_0|{𝐱}^{\left(n\right)}\right)$	0.00
$\mathrm{Pr}\left(M_1|{𝐱}^{\left(n\right)}\right)$	0.00
$\mathrm{Pr}\left(M_2|{𝐱}^{\left(n\right)}\right)$	1.00

Equations

$$\pi (m)$$

$$D_{KL}(f(\cdot |m)\Vert f(\cdot |m^{\prime }))=\int {\text{displaylimits}}_{\mathcal{X}}f(x|m)\cdot \log \left[\frac{f(x|m)}{f(x|m^{\prime })}\right]\mathrm{d}x.$$

$$M_j=\{f_j(x|{\theta }_j),{\pi }_j({\theta }_j)\}j\in \{1,2,\dots ,k\},$$

$$\mathrm{Pr}(M_j)\propto$$

$$\mathrm{Pr}(M_i|x)={\left[\sum _{j=1}^k\frac{\mathrm{Pr}(M_j)}{\mathrm{Pr}(M_i)}{B}_{ji}\right]}^{-1},$$

$$B_{ji}=\frac{\int f_j(x|{\theta }_j){\pi }_j({\theta }_j)\mathrm{d}{\theta }_j}{\int f_i(x|{\theta }_i){\pi }_i({\theta }_i)\mathrm{d}{\theta }_i},$$

$$X_1,\dots ,X_m|{\tilde{\theta }}_1$$

$$X_{m+1},\dots ,X_n|{\tilde{\theta }}_2$$

$$f({\mathbf{x}}^{(n)}|m,{\tilde{\theta }}_1,{\tilde{\theta }}_2)=\prod _{i=1}^m{f}_1(x_i|{\tilde{\theta }}_1)\prod _{i=m+1}^n{f}_2(x_i|{\tilde{\theta }}_2).$$

$$D_{KL}(f({\mathbf{x}}^{(n)}|m,{\tilde{\theta }}_1,{\tilde{\theta }}_2)\Vert f({\mathbf{x}}^{(n)}|m^{\prime },{\tilde{\theta }}_1,{\tilde{\theta }}_2))=$$

$$\log \left(\frac{f({\mathbf{x}}^{(n)}|m,{\tilde{\theta }}_1,{\tilde{\theta }}_2)}{f({\mathbf{x}}^{(n)}|m^{\prime },{\tilde{\theta }}_1,{\tilde{\theta }}_2)}\right)\mathrm{d}{\mathbf{x}}^{(n)}.$$

$$\frac{f({\mathbf{x}}^{(n)}|m,{\tilde{\theta }}_1,{\tilde{\theta }}_2)}{f({\mathbf{x}}^{(n)}|m^{\prime },{\tilde{\theta }}_1,{\tilde{\theta }}_2)}=\prod _{i=m+1}^{m^{\prime }}\frac{f_2(x_i|{\tilde{\theta }}_2)}{f_1(x_i|{\tilde{\theta }}_1)},$$

$$D_{KL}(f({\mathbf{x}}^{(n)}|m,{\tilde{\theta }}_1,{\tilde{\theta }}_2)$$

$$\sum _{i=m+1}^{m^{\prime }}\int f_2(x_i|{\tilde{\theta }}_2)\log \left(\frac{f_2(x_i|{\tilde{\theta }}_2)}{f_1(x_i|{\tilde{\theta }}_1)}\right)\mathrm{d}{x}_i.$$

$$D_{KL}(f({\mathbf{x}}^{(n)}|m,{\tilde{\theta }}_1,{\tilde{\theta }}_2)$$

$$(m^{\prime }-m)D_{KL}(f_2(\cdot |{\tilde{\theta }}_2)\Vert f_1(\cdot |{\tilde{\theta }}_1)).$$

$$D_{KL}(f({\mathbf{x}}^{(n)}|m,{\tilde{\theta }}_1,{\tilde{\theta }}_2)$$

$$(m-m^{\prime })D_{KL}(f_1(\cdot |{\tilde{\theta }}_1)\Vert f_2(\cdot |{\tilde{\theta }}_2)).$$

$$D_{KL}(f({\mathbf{x}}^{(n)}|m,{\tilde{\theta }}_1,{\tilde{\theta }}_2)\Vert f({\mathbf{x}}^{(n)}$$

$$\displaystyle\left\{\begin{array}[]{ll}l\cdot D_{KL}(f_{2}(\cdot|\tilde{\theta}_{2})\|f_{1}(\cdot|\tilde{\theta}_{1})),&\mbox{ if }m<m^{\prime}\\ &\\ l\cdot D_{KL}(f_{1}(\cdot|\tilde{\theta}_{1})\|f_{2}(\cdot|\tilde{\theta}_{2})),&\mbox{ if }m>m^{\prime},\end{array}\right.$$

$$\underset{m^{\prime }\ne m}{\min }$$

$$=\underset{m^{\prime }\ne m}{\min }\{l\cdot D_{KL}(f_2(\cdot |{\tilde{\theta }}_2)\Vert f_1(\cdot |{\tilde{\theta }}_1)),l\cdot D_{KL}(f_1(\cdot |{\tilde{\theta }}_1)\Vert f_2(\cdot |{\tilde{\theta }}_2))\}$$

$$=\underset{m^{\prime }\ne m}{\min }\{D_{KL}(f_2(\cdot |{\tilde{\theta }}_2)\Vert f_1(\cdot |{\tilde{\theta }}_1)),D_{KL}(f_1(\cdot |{\tilde{\theta }}_1)\Vert f_2(\cdot |{\tilde{\theta }}_2))\}\cdot \underset{1}{\underbrace{\underset{m^{\prime }\ne m}{\min }\{l\}}}.$$

$$\pi (m)=\frac{1}{n-1}m\in \{1,\dots ,n-1\}.$$

$$f({\mathbf{x}}^{(n)}|\mathbf{m},\tilde{\mathbf{\theta }})=\prod _{i=1}^{m_1}{f}_1(x_i|{\tilde{\theta }}_1)\prod _{j=1}^{k-1}\prod _{i=m_j+1}^{m_{j+1}}{f}_{j+1}(x_i|{\tilde{\theta }}_{j+1})\prod _{i=m_k+1}^n{f}_{k+1}(x_i|{\tilde{\theta }}_{k+1}),$$

$$\begin{array}[]{lcll}X_{1}&,\ldots,&X_{m_{1}}|\tilde{\theta}_{1}&\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny i.i.d.}}}{\sim}}f_{1}(\cdot|\tilde{\theta}_{1})\\ X_{m_{1}+1}&,\ldots,&X_{m_{2}}|\tilde{\theta}_{2}&\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny i.i.d.}}}{\sim}}f_{2}(\cdot|\tilde{\theta}_{2})\\ \vdots&,\ldots,&\vdots&\vdots\ldots\vdots\\ X_{m_{k-1}+1}&,\ldots,&X_{m_{k}}|\tilde{\theta}_{k}&\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny i.i.d.}}}{\sim}}f_{k}(\cdot|\tilde{\theta}_{k})\\ X_{m_{k}+1}&,\ldots,&X_{n}|\tilde{\theta}_{k+1}&\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny i.i.d.}}}{\sim}}f_{k+1}(\cdot|\tilde{\theta}_{k+1}).\\ \end{array}$$

$$d_j^{+1}(\tilde{\mathbf{\theta }})$$

$$d_j^{-1}(\tilde{\mathbf{\theta }})$$

$$D_{KL}(f({\mathbf{x}}^{(n)}|\mathbf{m},\tilde{\mathbf{\theta }})\Vert f({\mathbf{x}}^{(n)}|{\mathbf{m}}^{\prime },\tilde{\mathbf{\theta }})=|m_j^{\prime }-m_j|d_j^S(\tilde{\mathbf{\theta }}),$$

$$\pi (\mathbf{m})$$

$$\theta_{k}=\left\{\begin{array}[]{ll}\tilde{\theta}_{1},\ldots,\tilde{\theta}_{k+1},m_{1},\ldots,m_{k}&\mbox{ if }k=1,\dots,n-1\\ \tilde{\theta}_{1}&\mbox{ if }k=0\end{array}\right.$$

$$D_{KL}(M_i\Vert M_j)=D_{KL}(f({\mathbf{x}}^{(n)}|{\theta }_i)\Vert f({\mathbf{x}}^{(n)}|{\theta }_j)).$$

$$D_{KL}(M_i\Vert M_j)=\sum _{q=i+1}^j\left[(m_{q+1}-m_q)\cdot D_{KL}(f_{i+1}(\cdot |{\tilde{\theta }}_{i+1})\Vert f_{q+1}(\cdot |{\tilde{\theta }}_{q+1}))\right],$$

$$D_{KL}(M_j\Vert M_i)=\sum _{q=i+1}^j\left[(m_{q+1}-m_q)\cdot D_{KL}(f_{q+1}(\cdot |{\tilde{\theta }}_{q+1})\Vert f_{i+1}(\cdot |{\tilde{\theta }}_{i+1}))\right].$$

$$\mathrm{Pr}(M_j)\propto$$

Full text

Keywords: Change point; Discrete parameter space; Loss-based prior; Model selection.

1 Introduction

There are several practical scenarios where it is inappropriate to assume that the distribution of the observations does not change. For example, financial data sets can exhibit alternate behaviours due to crisis periods. In this case it is sensible to assume changes in the underlying distribution. The change in the distribution can be either in the value of one or more of the parameters or, more in general, on the family of the distribution. In the latter case, for example, one may deem appropriate to consider a normal density for the stagnation periods, while a Student $t$, with relatively heavy tails, may be more suitable to represent observations in the more turbulent stages of a crisis. The task of identifying if, and when, one or more changes have occurred is not trivial and requires appropriate methods to avoid detection of a large number of changes or, at the opposite extreme, seeing no changes at all. The change point problem has been deeply studied from a Bayesian point of view. Chernoff and Zacks (1964) focused on the change in the means of normally distributed variables. Smith (1975) looked into the single change point problem when different knowledge of the parameters of the underlying distributions is available: all known, some of them known or none of them known. Smith (1975) focuses on the binomial and normal distributions. In Muliere and Scarsini (1985) the problem is tackled from a Bayesian nonparametric perspective. The authors consider Dirichlet processes with independent base measures as underlying distributions. In this framework, Petrone and Raftery (1997) have showed that the Dirichlet process prior could have a strong effect on the inference and may lead to wrong conclusions in the case of a single change point. Raftery and Akman (1986) have approached the single change point problem in the context of a Poisson likelihood under both proper and improper priors for the model parameters. Carlin et al. (1992) build on the work of Raftery and Akman (1986) by considering a two level hierarchical model. Both papers illustrate the respective approaches by studying the well-known British coal-mining disaster data set. In the context of multiple change points detection, Loschi and Cruz (2005) have provided a fully Bayesian treatment for the product partitions model of Barry and Hartigan (1992). Their application focused on stock exchange data. Stephens (1994) has extended the Gibbs sampler introduced by Carlin et al. (1992) in the change point literature to handle multiple change points. Hannart and Naveau (2009) have used Bayesian decision theory, in particular 0-1 cost functions, to estimate multiple changes in homoskedastic normally distributed observations. Schwaller and Robin (2017) extend the product partition model of Barry and Hartigan (1992) by adding a graphical structure which could capture the dependencies between multivariate observations. Fearnhead and Liu (2007) proposed a filtering algorithm for the sequential multiple change points detection problem in the case of piecewise regression models. Henderson and Matthews (1993) introduced a partial Bayesian approach which involves the use of a profile likelihood, where the aim is to detect multiple changes in the mean of Poisson distributions with an application to haemolytic uraemic syndrome (HUS) data. The same data set was studied by Tian et al. (2009), who proposed a method which treats the change points as latent variables. Ko et al. (2015) have proposed an extension to the hidden Markov model of Chib (1998) by using a Dirichlet process prior on each row of the regime matrix. Their model is semiparametric, as the number of states is not specified in advance, but it grows according to the data size. Heard and Turcotte (2017) have proposed a new sequential Monte Carlo algorithm to infer multiple change points.

Whilst the literature covering change point analysis from a Bayesian perspective is vast when prior distributions are elicited, the documentation referring to analysis under minimal prior information is limited, see Moreno et al. (2005) and Girón et al. (2007). The former paper discusses the single change point problem in a model selection setting, whilst the latter paper, which is an extension of the former, tackles the multivariate change point problem in the context of linear regression models. Our work aims to contribute to the methodology for change point analysis under the assumption that the information about the number of change points and their location is minimal. First, we discuss the definition of an objective prior for change point location, both for single and multiple changes, assuming the number of changes is known a priori. Then, we define a prior on the number of change points via a model selection approach. Here, we assume that the change point coincides with one of the observations. As such, given $X_{1},X_{2},\dots,X_{n}$ data points, the change point location is discrete. To the best of our knowledge, the sole general objective approach to define prior distributions on discrete spaces is the one introduced by Villa and Walker (2015a).

To illustrate the idea, consider a probability distribution $f(x|m)$, where $m\in\mathbb{M}$ is a discrete parameter. Then, the prior $\pi(m)$ is obtained by objectively measuring what is lost if the value $m$ is removed from the parameter space, and it is the true value. According to Berk (1966), if a model is misspecified, the posterior distribution asymptotically accumulates on the model which is the most similar to the true one, where the similarity is measured in terms of the Kullback–Leibler (KL) divergence. Therefore, $D_{KL}(f(\cdot|m)\|f(\cdot|m^{\prime}))$, where $m^{\prime}$ is the parameter characterising the nearest model to $f(x|m)$, represents the utility of keeping $m$. The objective prior is then obtained by linking the aforementioned utility via the self-information loss:

[TABLE]

where the Kullback–Leibler divergence (Kullback and Leibler, 1951) from the sampling distribution with density $f(x|m)$ to the one with density $f(x|m^{\prime})$ is defined as:

[TABLE]

Throughout the paper, the objective prior defined in equation (1) will be referenced as the loss-based prior. This approach is used to define an objective prior distribution when the number of change points is known a priori. To obtain a prior distribution for the number of change points, we adopt a model selection approach based on the results in Villa and Walker (2015b), where a method to define a prior on the space of models is proposed. To illustrate, let us consider $k$ Bayesian models:

[TABLE]

where $f_{j}(x|\theta_{j})$ is the sampling density characterised by $\theta_{j}$ and $\pi_{j}(\theta_{j})$ represents the prior on the model parameter.

Assuming the prior on the model parameter, $\pi_{j}(\theta_{j})$, is proper, the model prior probability $\Pr(M_{j})$ is proportional to the expected minimum Kullback–Leibler divergence from $M_{j}$, where the expectation is considered with respect to $\pi_{j}(\theta_{j})$. That is:

[TABLE]

The model prior probabilities defined in equation (3) can be employed to derive the model posterior probabilities through:

[TABLE]

where $B_{ji}$ is the Bayes factor between model $M_{j}$ and model $M_{i}$, defined as

[TABLE]

with $i\neq j\in\{1,2,\ldots,k\}$.

This paper is structured as follows: in Section 2 we establish the way we set objective priors on both single and multiple change point locations. Section 3 shows how we define the model prior probabilities for the number of change point locations. Illustrations of the model selection exercise are provided in Sections 4 and 5, where we work with simulated and real data, respectively. Section 6 is dedicated to final remarks.

2 Objective Prior on the Change Point Locations

This section is devoted to the derivation of the loss-based prior when the number of change points is known a priori. Specifically, let $k$ be the number of change points and $m_{1}<m_{2}<\ldots<m_{k}$ their locations. We introduce the idea in the simple case where we assume that there is only one change point in the data set (see Section 2.1). Then, we extend the results to the more general case where multiple change points are assumed (see Section 2.2).

A well-known objective prior for finite parameter spaces, in cases where there is no structure, is the uniform prior (Berger et al., 2012). As such, a natural choice for the prior on the change points location is the uniform (Koop and Potter, 2009). The corresponding loss-based prior is indeed the uniform, as shown below, which is a reassuring result as the objective prior for a specific parameter space, if exists, should be unique.

2.1 Single Change Point

As mentioned above, we show that the loss-based prior for the single change point case coincides with the discrete uniform distribution over the set $\{1,2,\ldots,n-1\}$.

Let $\mathbf{X}^{(n)}=(X_{1},\ldots,X_{n})$ denote an n-dimensional vector of random variables, representing the random sample, and $m$ be our single change point location, that is $m\in\{1,2,\ldots,n-1\}$, such that

[TABLE]

Note that we assume that there is a change point in the series, as such the space of $m$ does not include the case $m=n$. In addition, we assume that $\tilde{\theta}_{1}\neq\tilde{\theta}_{2}$ when $f_{1}=f_{2}$. The sampling density for the vector of observations $\mathbf{x}^{(n)}=(x_{1},\dots,x_{n})$ is:

[TABLE]

Let $m^{\prime}\neq m$. Then, the Kullback–Leibler divergence between the model parametrised by $m$ and the one parametrised by $m^{\prime}$ is:

[TABLE]

Without loss of generality, consider $m<m^{\prime}$. In this case, note that

[TABLE]

leading to

[TABLE]

On the right hand side of equation (8), we can recognise the Kullback–Leibler divergence from density $f_{2}$ to density $f_{1}$, thus getting:

[TABLE]

In a similar fashion, when $m>m^{\prime}$, we have that:

[TABLE]

In this single change point scenario, we can consider $m^{\prime}$ as a perturbation of the change point location $m$, that is $m^{\prime}=m\pm l$ where $l\in\mathbb{N}^{*}$, such that $1\leq m^{\prime}<n$. Then, taking into account equations (9) and (10), the Kullback–Leibler divergence becomes:

[TABLE]

and

[TABLE]

We observe that equation (11) is only a function of $\tilde{\theta}_{1}$ and $\tilde{\theta}_{2}$ and does not depend on $m$. Thus, $\pi(m)\propto 1$ and, therefore,

[TABLE]

This prior was used, for instance, in an econometric context by Koop and Potter (2009) with the rationale of giving equal weight to every possible change point location.

2.2 Multivariate Change Point Problem

In this section, we address the change point problem in its generality by assuming that there are $1\leq k<n$ change points. In particular, for the data $\mathbf{x}^{(n)}=(x_{1},\dots,x_{n})$, we consider the following sampling distribution

[TABLE]

where $\bm{m}=(m_{1},\dots,m_{k})$, $1\leq m_{1}<m_{2}<\ldots<m_{k}<n$, is the vector of the change point locations and $\bm{\tilde{\theta}}=(\tilde{\theta}_{1},\dots,\tilde{\theta}_{k},\tilde{\theta}_{k+1})$ is the vector of the parameters of the underlying probability distributions. Schematically:

[TABLE]

If $f_{1}=f_{2}=\cdots=f_{k+1}$, then it is reasonable to assume that some of the $\theta$’s are different. Without loss of generality, we assume that $\tilde{\theta}_{1}\neq\tilde{\theta}_{2}\neq\cdots\neq\tilde{\theta}_{k}\neq\tilde{\theta}_{k+1}$. In a similar fashion to the single change point case, we cannot assume $m_{k}=n$ since we require exactly $k$ change points.

In this case, due to the multivariate nature of the vector $\bm{m}=(m_{1},\dots,m_{k})$, the derivation of the loss-based prior is not as straightforward as in the one dimensional case. In fact, the derivation of the prior is based on heuristic considerations supported by the below Theorem 1 (the proof of which is in the Appendix). In particular, we are able to prove an analogous of equations (9) and (10) when only one component is arbitrarily perturbed. Let us define the following functions:

[TABLE]

where $j\in\{1,2,\ldots,k\}$. The following Theorem is useful to understand the behaviour of the loss-based prior in the general case.

Theorem 1**.**

Let $f(\mathbf{x}^{(n)}|\bm{m},\bm{\tilde{\theta}})$ be the sampling distribution defined in equation (13) and consider $j\in\{1,\dots,k\}$. Let $\bm{m}^{\prime}$ be such that $m^{\prime}_{i}=m_{i}$ for $i\neq j$, and let the component $m^{\prime}_{j}$ be such that $m^{\prime}_{j}\neq m_{j}$ and $m_{j-1}<m^{\prime}_{j}<m_{j+1}$. Therefore,

[TABLE]

where $S=\emph{sgn}(m^{\prime}_{j}-m_{j})$.

Note that, Theorem 1 states that the minimum Kullback–Leibler divergence is achieved when $m^{\prime}_{j}=m_{j}+1$ or $m^{\prime}_{j}=m_{j}-1$. This result is not surprising since the Kullback–Leibler divergence measures the degree of similarity between two distributions. The smaller the perturbation caused by changes in one of the parameters is, the smaller the Kullback–Leibler divergence between the two distributions is. Although Theorem 1 makes a partial statement about the multiple change points scenario, it provides a strong argument for supporting the uniform prior. Indeed, if now we consider the general case of having $k$ change points, it is straightforward to see that the Kullback–Leibler divergence is minimised when only one of the components of the vector $\bm{m}$ is perturbed by (plus or minus) one unit. As such, the loss-based prior depends on the vector of parameters $\bm{\tilde{\theta}}$ only, as in the one-dimensional case, yielding the uniform prior for $\bm{m}$.

Therefore, the loss-based prior on the multivariate change point location is

[TABLE]

where $\bm{m}=(m_{1},\dots,m_{k}),1\leq m_{1}<m_{2}<\ldots<m_{k}<n$. The denominator in equation (14) has the above form because, for every number of $k$ change points, we are interested in the number of $k$-subsets from a set of $n-1$ elements, which is $\binom{n-1}{k}$. The same prior was also derived in a different way by Girón et al. (2007).

3 Loss-based Prior on the Number of Change Points

Here, we approach the change point analysis as a model selection problem. In particular, we define a prior on the space of models, where each model represents a certain number of change points (including the case of no change points). The method adopted to define the prior on the space of models is the one introduced in Villa and Walker (2015b).

We proceed as follows. Assume we have to select from $k+1$ possible models. Let $M_{0}$ be the model with no change points, $M_{1}$ the model with one change point and so on. Generalising, model $M_{k}$ corresponds to the model with $k$ change points. The idea is that the current model encompasses the change point locations of the previous model. As an example, in model $M_{3}$ the first two change point locations will be the same as in the case of model $M_{2}$. To illustrate the way we envision our models, we have provided Figure 1. It has to be noted that the construction of the possible models from $M_{0}$ to $M_{k}$ can be done in a different way to one here described. Obviously, the approach to define the model priors stays unchanged. Consistently with the notation used in Section 1,

[TABLE]

represents the vector of parameters of model $M_{k}$, where $\tilde{\theta}_{1},\ldots,\tilde{\theta}_{k+1}$ are the model specific parameters and $m_{1},\ldots,m_{k}$ are the change point locations, as in Figure 1.

Based on the way we have specified our models, which are in direct correspondence with the number of change points and their locations, we state Theorem 2 (the proof of which is in the Appendix).

Theorem 2**.**

Let

[TABLE]

For any $0\leq i<j\leq k$ integers, with $k<n$, and the convention $m_{j+1}=n$, we have the following:

[TABLE]

and

[TABLE]

The result in Theorem 2 is useful when the model selection exercise is implemented. Indeed, the Villa and Walker (2015b) approach requires the computation of the Kullback–Leibler divergences in Theorem 2. Recalling equation (3), the objective model prior probabilities are then given by:

[TABLE]

For illustrative purposes, in the Appendix we derive the model prior probabilities to perform model selection among $M_{0}$, $M_{1}$ and $M_{2}$.

It is easy to infer from equation (15) that model priors depend on the prior distribution assigned to the model parameters, that is on the level of uncertainty that we have about their true values. For the change point location, a sensible choice is the uniform prior which, as shown in Section 2, corresponds to the loss-based prior. For the model specific parameters, we have several options. If one wishes to pursue an objective analysis, intrinsic priors (Berger and Pericchi, 1996) may represent a viable solution since they are proper. Nonetheles, the method introduce by Villa and Walker (2015b) does not require, in principle, an objective choice as long as the priors are proper. Given that we use the latter approach, here we consider subjective priors for the model specific parameters.

Remark. In the case where the changes in the underlying sampling distribution are limited to the parameter values, the model prior probabilities defined in (15) follow the uniform distribution. That is, $\Pr(M_{j})\propto 1$. In the real data example illustrated in Section 5.1, we indeed consider a problem where the above case occurs.

3.1 A special case: selection between $M_{0}$ and $M_{1}$

Let us consider the case where we have to estimate whether there is or not a change point in a set of observations. This implies that we have to choose between model $M_{0}$ (i.e. no change point) and $M_{1}$ (i.e. one change point). Following our approach, we have:

[TABLE]

and

[TABLE]

Now, let us assume independence between the prior on the change point location and the prior on the parameters of the underlying sampling distributions, that is $\pi_{1}(m_{1},\tilde{\theta}_{1},\tilde{\theta}_{2})=\pi_{1}(m_{1})\pi_{1}(\tilde{\theta}_{1},\tilde{\theta}_{2})$. Let us further recall that, as per equation (14), $\pi_{1}(m_{1})=1/(n-1)$. As such, we observe that the model prior probability on $M_{1}$ becomes:

[TABLE]

We notice that the model prior probability for model $M_{1}$ is increasing when the sample size increases. This behaviour occurs whether there is or not a change point in the data. We propose to address the above problem by using a non-uniform prior for $m_{1}$. A reasonable alternative, which works quite well in practice, would be the following shifted binomial as prior:

[TABLE]

To argument the choice of (19), we note that, as $n$ increases, the probability mass will be more and more concentrated towards the upper end of the support. Therefore, from equations (17) and (19) follows:

[TABLE]

For the more general case where we consider more than two models, the problem highlighted in equation (18) vanishes.

4 Change Point Analysis on Simulated Data

In this section, we present the results of several simulation studies based on the methodologies discussed in Sections 2 and 3. We start with a scenario involving discrete distributions in the context of the one change point problem. We then show the results obtained when we consider continuous distributions for the case of two change points. The choice of the underlying sampling distributions is in line with Villa and Walker (2015b).

4.1 Single sample

Scenario 1.

The first scenario concerns the choice between models $M_{0}$ and $M_{1}$. Specifically, for $M_{0}$ we have:

[TABLE]

and for $M_{1}$ we have:

[TABLE]

Let us denote with $f_{1}(\cdot|p)$ and $f_{2}(\cdot|\lambda)$ the probability mass functions of the Geometric and the Poisson distributions, respectively. The priors for the parameters of $f_{1}$ and $f_{2}$ are $p\sim\mbox{Beta}(a,b)$ and $\lambda\sim\mbox{Gamma}(c,d)$.

In the first simulation, we sample $n=100$ observations from model $M_{0}$ with $p=0.8$. To perform the change point analysis, we have chosen the following parameters for the priors on $p$ and $\lambda$: $a=2$, $b=2$, $c=3$ and $d=1$. Applying the approach introduced in Section 3, we obtain $\Pr(M_{0})\propto 1.59$ and $\Pr(M_{1})\propto 1.81$. These model priors yield the posterior distribution probabilities (refer to equation (4)) $\Pr(M_{0}|\mathbf{x}^{(n)})=0.92$ and $\Pr(M_{1}|\mathbf{x}^{(n)})=0.08$. As expected, the selection process strongly indicates the true model as $M_{0}$. Table 1 reports the above probabilities including other information, such as the appropriate Bayes factors.

The second simulation looked at the opposite setup, that is we sample $n=100$ observations from $M_{1}$, with $p=0.8$ and $\lambda=3$. We have sampled 50 data points from the Geometric distribution and the remaining 50 data points from the Poisson distribution. In Figure 2, we have plotted the simulated sample, where it is legitimate to assume a change in the underlying distribution. Using the same prior parameters as above, we obtain $\Pr(M_{0}|\mathbf{x}^{(n)})=0.06$ and $\Pr(M_{1}|\mathbf{x}^{(n)})=0.94$. Again, the model selection process is assigning heavy posterior mass to the true model $M_{1}$. These results are further detailed in Table 1.

Scenario 2.

In this scenario we consider the case where we have to select among three models, that is model $M_{0}$:

[TABLE]

model $M_{1}$:

[TABLE]

with $1\leq m_{1}\leq n-1$ being the location of the single change point, and model $M_{2}$:

[TABLE]

with $1\leq m_{1}<m_{2}\leq n-1$ representing the locations of the two change points, such that $m_{1}$ corresponds exactly to the same location as in model $M_{1}$. Analogously to the previous scenario, we sample from each model in turn and perform the selection to detect the number of change points.

Let $f_{1}(\cdot|\lambda,\kappa),f_{2}(\cdot|\mu,\tau)$ and $f_{3}(\cdot|\alpha,\beta)$ represent the Weibull, Log-normal and Gamma densities, respectively, with $\tilde{\theta}_{1}=(\lambda,\kappa)$, $\tilde{\theta}_{2}=(\mu,\tau)$ and $\tilde{\theta}_{3}=(\alpha,\beta)$. We assume a Normal prior on $\mu$ and Gamma priors on all the other parameters as follows:

[TABLE]

[TABLE]

In the first exercise, we have simulated $n=100$ observations from model $M_{0}$, where we have set $\lambda=1.5$ and $\kappa=5$. We obtain the following model priors: $\Pr(M_{0})\propto 1.09$, $\Pr(M_{1})\propto 1.60$ and $\Pr(M_{2})\propto 1.37$, yielding the posteriors $\Pr(M_{0}|\mathbf{x}^{(n)})=0.96$, $\Pr(M_{1}|\mathbf{x}^{(n)})=0.04$ and $\Pr(M_{2}|\mathbf{x}^{(n)})=0.00$. We then see that the approach assigns high mass to the true model $M_{0}$. Table 2 reports the above probabilities and the corresponding Bayes factors.

The second simulation was performed by sampling 50 observations from a Weibull with parameter values as in the previous exercise, and the remaining 50 observations from a Log-normal density with location parameter $\mu=0.05$ and scale parameter $\tau=16$. The data is displayed in Figure 3.

The model posterior probabilities are $\Pr(M_{0}|\mathbf{x}^{(n)})=0.00$, $\Pr(M_{1}|\mathbf{x}^{(n)})=0.98$ and $\Pr(M_{2}|\mathbf{x}^{(n)})=0.02$, which are reported in Table 2. In this case as well, we see that the model selection procedure indicates $M_{1}$ as the true model, as expected.

Finally, for the third simulation exercise we sample 50 and 20 data points from, respectively, a Weibull and a Log-normal with parameter values as defined above, and the last 30 observations are sampled from a Gamma distribution with parameters $\alpha=10$ and $\beta=2$. From Table 2, we note that the posterior distribution on the model space accumulates on the true model $M_{2}$.

4.2 Frequentist Analysis

In this section, we perform a frequentist analysis of the performance of the proposed prior by drawing repeated samples from different scenarios. In particular, we look at a two change points problem where the sampling distributions are Student-$t$ with different degrees of freedom. In this scenario, we perform the analysis with $60$ repeated samples generated by different densities with the same mean values.

Then, we repeat the analysis of Scenario 2 by selecting $100$ samples for $n=500$ and $n=1500$. We consider different sampling distributions with the same mean and variance. In this scenario, where we added the further constraint of the equal variance, it is interesting to note that the change in distribution is captured when we increase the sample size, meaning that we learn more about the true sampling distributions.

We also compare the performances of the loss-based prior with the uniform prior when we analyse the scenario with different sampling distributions. Namely, Weibull/Log-normal/Gamma. It is interesting to note that the uniform prior is unable to capture the change in distribution even for a large sample size. On the contrary, the loss-based prior is able to detect the number of change points when $n=1500$. Furthermore, for $n=500$, even though both priors are not able to detect the change points most of the times, the loss-based prior has a higher frequency of success when compared to the uniform prior.

Scenario 3.

In this scenario, we consider the case where the sampling distributions belong to the same family, that is Student-$t$, where the true model has two change points. In particular, let $f_{1}(\cdot|\nu_{1}),f_{2}(\cdot|\nu_{2})$ and $f_{3}(\cdot|\nu_{3})$ represent the densities of three standard $t$ distributions, respectively. We assume that $\nu_{1},\nu_{2}$ and $\nu_{3}$ are positive integers strictly greater than one so to have defined mean for each density. Note that this allows us to compare distributions of the same family with equal mean. The priors assigned to the number of degrees of freedom assume a parameter space of positive integers strictly larger than 1. As such, we define them as follows:

[TABLE]

In this experiment, we consider 60 repeated samples, each of size $n=300$ and with the following structure:

• •

$X_{1},\dots,X_{100}$ from a Student-$t$ distribution with $\nu_{1}=30$,

• •

$X_{101},\dots,X_{200}$ from a Student-$t$ distribution with $\nu_{2}=3$,

• •

$X_{201},\dots,X_{300}$ from a Student-$t$ distribution with $\nu_{3}=8$.

Table 3 reports the frequentist results of the simulation study. First, note that $P(M_{1})=P(M_{2})=P(M_{3})=1/3$ as per the Remark in Section 3. For all the simulated samples, the loss-based prior yields a posterior with the highest probability assigned to the true model $M_{2}$. We also note that the above posterior is on average $0.75$ with a variance $0.02$, making the inferential procedure extremely accurate.

Scenario 4.

In this scenario, we perform repeated sampling from the setup described in scenario 2 above, where the true model has two change points. In particular, we draw $100$ samples with $n=500$ and $n=1500$. For $n=500$, the loss-based prior probabilities are $P(M_{0})=0.18$, $P(M_{1})=0.16$ and $P(M_{2})=0.66$. For $n=1500$, the loss-based prior probabilities are $P(M_{0})=0.015$, $P(M_{1})=0.014$ and $P(M_{2})=0.971$. The simulation results are reported, respectively, in Table 4 and in Table 5. The two change point locations for $n=500$ are at the $171$st and $341$st observations. For $n=1500$, the first change point is the $501$st observation, while the second is at the $1001$st observation. We note that there is a sensible improvement in detecting the true model, using the loss-based prior, when the sample size increases. In particular, we move from $30\%$ to $96\%$.

To compare the loss-based prior with the uniform prior we have run the simulation on the same data samples used above. The results for $n=500$ and $n=1500$ are in Table 6 and in Table 7, respectively. Although we can observe an improvement when the sample size increases, the uniform prior does not lead to a clear detection of the true model for both sample sizes.

Finally, we conclude this section with a remark. One may wonder why the change point detection requires an increasing in the sample size, and the reply can be inferred from Figure 5, which displays the density functions of the distributions employed in this scenario. As it can be observed, the densities are quite similar, which is not surprising since these distributions have the same means and the same variances. The above similarity can also be appreciated in terms of Hellinger distance, see Table 8. In other words, from Figure 5 we can see that the main differences in the underlying distributions are in the tail areas. It is therefore necessary to have a relatively large number of observations in order to be able to discern differences in the densities, because in this case only we would have a sufficient representation of the whole distribution.

5 Change Point Analysis on Real Data

In this section, we illustrate the proposed approach applied to real data. We first consider a well known data set which has been extensively studied in the literature of the change point analysis, that is the British coal-mining disaster data (Carlin et al., 1992). The second set of data we consider refers to the daily returns of the S&P 500 index observed over a period of four years. The former data set will be investigated in Section 5.1, while the latter in Section 5.2.

5.1 British Coal-Mining Disaster Data

The British coal-mining disaster data consists of the yearly number of deaths for the British coal miners over the period 1851-1962. It is believed that the change in the working conditions, and in particular, the enhancement of the security measures, led to a decrease in the number of deaths. This calls for a model which can take into account a change in the underlying distribution around a certain observed year. With the proposed methodology we wish to detect if the assumption is appropriate. In particular, if a model with one change point is more suitable to represent the data than a model where no changes in the sampling distribution are assumed. Figure 6 shows the number of deaths per year in the British coal-mining industry from 1851 to 1962.

As in Chib (1998), we assume a Poisson sampling distribution with a possible change in the parameter value. That is

[TABLE]

where $m$ is the unknown location of the single change point, such that $1\leq m\leq n$, and a $\mbox{Gamma}(2,1)$ is assumed for $\phi_{1}$ and $\phi_{2}$. The case $m=n$ corresponds to the scenario with no change point, that is model $M_{0}$. The case $m<n$ assumes one change point, that is model $M_{1}$.

Let $f_{1}(\cdot|\phi_{1})$ and $f_{2}(\cdot|\phi_{2})$ be the Poisson distributions with parameters $\phi_{1}$ and $\phi_{2}$, respectively. Then, the analysis is performed by selecting between model $M_{0}$, that is when the sampling distribution is $f_{1}$, and model $M_{1}$, where the sampling distribution is $f_{1}$ up to a certain $m<n$ and $f_{2}$ from $m+1$ to $n$.

As highlighted in the Remark at the end of Section 3, the prior on the model space is the discrete uniform distribution, that is $\Pr(M_{0})=\Pr(M_{1})=0.5$. The proposed model selection approach leads to the Bayes factors $B_{01}=1.61\times 10^{-13}$ and $B_{10}=6.20\times 10^{12}$, where it is obvious that the odds are strongly in favour of model $M_{1}$. Indeed, we have $\Pr(M_{1}|{\bf x}^{(n)})\approx 1$.

5.2 Daily S&P 500 Absolute Log-Return Data

The second real data analysis aims to detect change points in the absolute value of the daily logarithmic returns of the S&P500 index observed from the 14/01/2008 to the 31/12/2011 (see Figure 7). As underlying sampling distributions we consider the Weibull and the Log-normal (Yu, 2001), and the models among which we select are as follows. $M_{0}$ is a $\mbox{Weibull}(\lambda,\kappa)$, $M_{1}$ is formed by a $\mbox{Weibull}(\lambda,\kappa)$ and a $\mbox{Log-normal}(\mu_{1},\tau_{1})$ and, finally, $M_{2}$ is formed by a $\mbox{Weibull}(\lambda,\kappa)$, a $\mbox{Log-normal}(\mu_{1},\tau_{1})$ and a $\mbox{Log-normal}(\mu_{2},\tau_{2})$. An interesting particularity of this problem is that we will consider a scenario where the changes are in the underlying distribution or in the parameter values of the same distribution. As suggested in Section 4.1.3 of Kass and Raftery (1995), due to the large sample size of the data set, we could approximate the Bayes factor by using the Schwarz criterion. Therefore, in this case the specification of the priors for the parameters of the underlying distributions is not necessary.

From the results in Table 9, we see that the model indicated by the proposed approach is $M_{2}$. In other words, there is very strong indication that there are two change points in the data set.

From Table 9, we note that the prior on model $M_{1}$ and $M_{2}$ assigned by the proposed method are the same. This is not surprising as the only difference between the two models is an additional Log-normal distribution with different parameter values.

6 Conclusion

Bayesian inference in change point problems under the assumption of minimal prior information has not been deeply explored in the past, as the limited literature on the matter shows.

We contribute to the area by deriving an objective prior distribution to detect change point locations, when the number of change points is known a priori. As a change point location can be interpreted as a discrete parameter, we apply recent results in the literature (Villa and Walker, 2015a) to make inference. The resulting prior distribution, which is the discrete uniform distribution, it is not new in the literature (Girón et al., 2007), and therefore can be considered as a validation of the proposed approach.

A second major contribution is in defining an objective prior on the number of change points, which has been approached by considering the problem as a model selection exercise. The results of the proposed method on both simulated and real data, show the strength of the approach in estimating the number of change points in a series of observations. A point to note is the generality of the scenarios considered. Indeed, we consider situations where the change is in the value of the parameter(s) of the underlying sampling distribution, or in the distribution itself. Of particular interest is the last real data analysis (S&P 500 index), where we consider a scenario where we have both types of changes, that is the distribution for the first change point and on the parameters of the distribution for the second.

The aim of this work was to set up a novel approach to address change point problems. In particular, we have selected prior densities for the parameters of the models to reflect a scenario of equal knowledge, in the sense that model priors are close to represent a uniform distribution. Two remarks are necessary here. First, in the case prior information about the true value of the parameters is available, and one wishes to exploit it, the prior densities will need to reflect it and, obviously, the model prior will be impacted by the choice. Second, in applications it is recommended that some sensitivity analysis is performed, so to investigate if and how the choice of the parameter densities affects the selection process.

Acknowledgements

Fabrizio Leisen was supported by the European Community’s Seventh Framework Programme [FP7/2007-2013] under grant agreement no: 630677. Cristiano Villa was supported by the Royal Society Research Grant no: RG150786.

Appendix

A Model prior probabilities to select among models $M_{0}$, $M_{1}$ and $M_{2}$

Here, we show how model prior probabilities can be derived for the relatively simple case of selecting among scenarios with no change points ($M_{0}$), one change point ($M_{1}$) or two change points ($M_{2}$). First, by applying the result in Theorem 2, we derive the Kullback–Leibler divergences between any two models. That is:

• •

the prior probability for model $M_{0}$ depends on the following quantities:

[TABLE]

• •

the prior probability for model $M_{1}$ depends on the following quantities:

[TABLE]

• •

the prior probability for model $M_{2}$ depends on the following quantities:

[TABLE]

The next step is to derive the minimum Kullback–Leibler divergence computed at each model:

• •

for model $M_{0}$:

[TABLE]

• •

for model $M_{1}$:

[TABLE]

• •

for model $M_{2}$:

[TABLE]

Therefore, the model prior probabilities can be computed through equation (15), so that:

• •

the model prior probability $\Pr(M_{0})$ is proportional to the exponential of the minimum between:

[TABLE]

• •

the model prior probability $\Pr(M_{1})$ is proportional to the exponential of the minimum between:

[TABLE]

• •

the model prior probability $\Pr(M_{2})$ is proportional to the exponential of the minimum between:

[TABLE]

B Proofs

Proof of Theorem 1

We distinguish two cases: $S=+1$ and $S=-1$. When $S=+1$, equivalent to $m_{j}<m_{j}^{\prime}$:

[TABLE]

When $S=-1$, equivalent to $m_{j}>m_{j}^{\prime}$, in a similar fashion, we get

[TABLE]

From equations (25) and (26), we get the result in Theorem 1.

Proof of Theorem 2

We recall that the model parameter $\theta_{i}$ is the vector $(m_{1},m_{2},\ldots,m_{i},\tilde{\theta}_{1},\tilde{\theta}_{2},\ldots,\tilde{\theta}_{i+1})$, where $i=0,1,\ldots,k$. Here, $\tilde{\theta}_{1},\tilde{\theta}_{2},\ldots,\tilde{\theta}_{i+1}$ represent the parameters of the underlying sampling distributions considered under model $M_{i}$ and $m_{1},m_{2},\ldots,m_{i}$ are the respective $i$ change point locations. In this setting,

[TABLE]

We proceed to the computation of $D_{KL}(M_{i}\|M_{j})$, that is the Kullback–Leibler divergence introduced in Section 3. Similarly to the proof of Theorem 1, we obtain the following result.

[TABLE]

Given equation (27), if we integrate out the variables not involved in the logarithms, we obtain

[TABLE]

In a similar fashion, it can be shown that

[TABLE]