Regularized divergences between covariance operators and Gaussian measures on Hilbert spaces

TL;DR

This paper extends divergences between Gaussian measures from finite-dimensional spaces to infinite-dimensional Hilbert spaces, providing explicit formulas and showing convergence of regularized divergences to true divergences.

Contribution

It introduces regularized Kullback-Leibler and Rényi divergences for Gaussian measures on Hilbert spaces using infinite-dimensional Alpha Log-Determinant divergences, with convergence results.

Findings

Explicit formulas for divergences in the Gaussian Hilbert space setting

Regularized divergences converge to true divergences as regularization vanishes

General Gaussian setting covered

Abstract

This work presents an infinite-dimensional generalization of the correspondence between the Kullback-Leibler and R\'enyi divergences between Gaussian measures on Euclidean space and the Alpha Log-Determinant divergences between symmetric, positive definite matrices. Specifically, we present the regularized Kullback-Leibler and R\'enyi divergences between covariance operators and Gaussian measures on an infinite-dimensional Hilbert space, which are defined using the infinite-dimensional Alpha Log-Determinant divergences between positive definite trace class operators. We show that, as the regularization parameter approaches zero, the regularized Kullback-Leibler and R\'enyi divergences between two equivalent Gaussian measures on a Hilbert space converge to the corresponding true divergences. The explicit formulas for the divergences involved are presented in the most general Gaussian setting.

Equations

$$d_{\mathrm{logdet}}^{\alpha }[(A+\gamma I),(B+\mu I)]$$

$$=\frac{4}{1-{\alpha }^2}\log \left[\frac{\mathrm{de}{\mathrm{t}}_{\mathrm{X}}\left(\frac{1-\alpha }{2}(A+\gamma I)+\frac{1+\alpha }{2}(B+\mu I)\right)}{\mathrm{de}{\mathrm{t}}_{\mathrm{X}}(A+\gamma I{)}^{\beta }\mathrm{de}{\mathrm{t}}_{\mathrm{X}}(B+\mu I{)}^{1-\beta }}{\left(\frac{\gamma }{\mu }\right)}^{\beta -\frac{1-\alpha }{2}}\right],$$

$$d_{\mathrm{logdet}}^1[(A+\gamma I),(B+\mu I)]=\left(\frac{\gamma }{\mu }-1\right)\log \frac{\gamma }{\mu }$$

$$+\mathrm{t}{\mathrm{r}}_{\mathrm{X}}[(B+\mu I{)}^{-1}(A+\gamma I)-I]-\frac{\gamma }{\mu }\log \mathrm{de}{\mathrm{t}}_{\mathrm{X}}[(B+\mu I{)}^{-1}(A+\gamma I)].$$

$$d_{\mathrm{logdet}}^{-1}[(A+\gamma I),(B+\mu I)]=\left(\frac{\mu }{\gamma }-1\right)\log \frac{\mu }{\gamma }$$

$$+\mathrm{t}{\mathrm{r}}_{\mathrm{X}}\left[(A+\gamma I{)}^{-1}(B+\mu I)-I\right]-\frac{\mu }{\gamma }\log \mathrm{de}{\mathrm{t}}_{\mathrm{X}}[(A+\gamma I{)}^{-1}(B+\mu I)].$$

$$d_{\mathrm{logdet}}^{\alpha }[(A+\gamma I),(B+\gamma I)]$$

$$d_{\mathrm{logdet}}^1[(A+\gamma I),(B+\gamma I)]$$

$$-\log \mathrm{de}{\mathrm{t}}_{\mathrm{X}}[(B+\gamma I{)}^{-1}(A+\gamma I)].$$

$$d_{\mathrm{logdet}}^{-1}[(A+\gamma I),(B+\gamma I)]$$

$$-\log \mathrm{de}{\mathrm{t}}_{\mathrm{X}}[(A+\gamma I{)}^{-1}(B+\gamma I)].$$

$$D_{\alpha }^{\gamma }(A,B)=d_{\mathrm{logdet}}^{\alpha }[(A+\gamma I),(B+\gamma I)],A,B\in {\mathrm{Sym}}^{+}(\mathcal{H})\cap \mathrm{Tr}(\mathcal{H}).$$

$$D_{\alpha }^{\gamma }[\mathcal{N}(0,C_1),\mathcal{N}(0,C_2)]=d_{\mathrm{logdet}}^{\alpha }[(C_1+\gamma I),(C_2+\gamma I)].$$

$$D_{\mathrm{KL}}^{\gamma }(\mathcal{N}(m_1,C_1)||\mathcal{N}(m_2,C_2))$$

$$+\frac{1}{2}{d}_{\mathrm{logdet}}^1[(C_1+\gamma I),(C_2+\gamma I)].$$

$$D_{\mathrm{R},r}^{\gamma }(\mathcal{N}(m_1,C_1)||\mathcal{N}(m_2,C_2))$$

$$=\frac{1}{2}⟨m_1-m_2,[(1-r)(C_1+\gamma I)+r(C_2+\gamma I){]}^{-1}(m_1-m_2)⟩$$

$$+\frac{1}{2}{d}_{\mathrm{logdet}}^{2r-1}[(C_1+\gamma I),(C_2+\gamma I)].$$

$$\underset{r\to 1}{\lim }{D}_{\mathrm{R},r}^{\gamma }(\mathcal{N}(m_1,C_1)||\mathcal{N}(m_2,C_2))=D_{\mathrm{KL}}^{\gamma }(\mathcal{N}(m_1,C_1)||\mathcal{N}(m_2,C_2)),$$

$$\underset{r\to 0}{\lim }{D}_{\mathrm{R},r}^{\gamma }(\mathcal{N}(m_1,C_1)||\mathcal{N}(m_2,C_2))=D_{\mathrm{KL}}^{\gamma }(\mathcal{N}(m_2,C_2)||\mathcal{N}(m_1,C_1)).$$

$$D_B^{\gamma }(\mathcal{N}(m_1,C_1),\mathcal{N}(m_2,C_2))$$

$$=\frac{1}{8}⟨(m_1-m_2),{\left(\frac{(C_1+\gamma I)+(C_2+\gamma I)}{2}\right)}^{-1}(m_1-m_2)⟩$$

$$+\frac{1}{8}{d}_{\mathrm{logdet}}^0[(C_1+\gamma I),(C_2+\gamma I)]=\frac{1}{4}{D}_{R,1/2}^{\gamma }(\mathcal{N}(m_1,C_1),\mathcal{N}(m_2,C_2)).$$

$$D_H^{\gamma }=\sqrt{2[1-\exp (-D_B^{\gamma })]}.$$

$$\underset{\gamma \to 0^{+}}{\lim }{D}_{\mathrm{KL}}^{\gamma }(\nu ||\mu )$$

$$=D_{\mathrm{KL}}(\nu ||\mu ),$$

$$\underset{\gamma \to 0^{+}}{\lim }{D}_{\mathrm{KL}}^{\gamma }(\nu ||\mu )$$

$$⟨x,y{⟩}_{C_0}=⟨C_0^{-1/2}x,C_0^{-1/2}y⟩,x,y\in \mathrm{Im}(C_0^{1/2})$$

$$\underset{\gamma \to 0^{+}}{\lim }{D}_{\mathrm{R},r}^{\gamma }(\nu ||\mu )$$

$$+\frac{1}{2r(1-r)}\log \det [(I-(1-r)S)(I-S{)}^{r-1}]$$

$$=D_{\mathrm{R},r}(\nu ||\mu ).$$

$$\underset{\gamma \to 0^{+}}{\lim }{D}_B^{\gamma }(\nu ||\mu )$$

$$+\frac{1}{2}\log \det [(I-\frac{1}{2}S)(I-S{)}^{-1/2}]$$

$$=D_B(\nu ||\mu ).$$

$$\underset{\gamma \to 0^{+}}{\lim }{D}_H^{\gamma }(\nu ||\mu )$$

$$=D_H(\nu ||\mu ).$$

$$y=Au+\eta ,\eta \sim \mathcal{N}(0,\Gamma ),\Gamma \in {\mathrm{Sym}}^{++}(n),$$

$$m=m_0+C_0{A}^{\ast }(\Gamma +A{C}_0{A}^{\ast }{)}^{-1}(y-A{m}_0),$$

$$C=C_0-C_0{A}^{\ast }(\Gamma +A{C}_0{A}^{\ast }{)}^{-1}A{C}_0.$$

$$D_{\mathrm{KL}}(\mathcal{N}(m,C)||\mathcal{N}(m_0,C_0))=\underset{\gamma \to 0^{+}}{\lim }{D}_{\mathrm{KL}}^{\gamma }(\mathcal{N}(m,C)||\mathcal{N}(m_0,C_0))$$

$$=\frac{1}{2}\left[\log \det (\Gamma +A{C}_0{A}^{\ast })-\log \det (\Gamma )-\mathrm{tr}(AC{A}^{\ast }{\Gamma }^{-1})-⟨m-m_0,A^{\ast }{\Gamma }^{-1}(Am-y)⟩\right].$$

Full text

1 Introduction

This work is concerned with the correspondence between divergences between covariance operators and the corresponding Gaussian measures on an infinite-dimensional Hilbert space. Specifically, we study the correspondence between the infinite-dimensional Alpha Log-Determinant (Log-Det) divergences between covariance operators on a Hilbert space $\mathcal{H}$ and the Kullback-Leibler and Rényi divergences, together with related quantities, between Gaussian measures on $\mathcal{H}$.

In the finite-dimensional setting, let ${\rm Sym}^{++}(n)$ denote the set of symmetric, positive definite (SPD) matrices. Then a divergence on ${\rm Sym}^{++}(n)$ correspond to a divergence on the set of zero-mean Gaussian measures on $\mathbb{R}^{n}$ with strictly positive covariance matrices. In particular, the Alpha Log-Det divergences Chebbi:2012Means on ${\rm Sym}^{++}(n)$ correspond to the Kullback-Leibler and Rényi divergences between zero-mean Gaussian measures on $\mathbb{R}^{n}$.

The infinite-dimensional generalization of the finite-dimensional setting requires substantially more mathematical machinery. It is not straightforward, for instance, to define Log-Determinant divergences between covariance operators on an infinite-dimensional Hilbert space $\mathcal{H}$, which are trace class operators, thus have vanishing eigenvalues and therefore unbounded inverses and principal logarithms. In Minh:LogDet2016 , the author generalized the Alpha Log-Det divergences on ${\rm Sym}^{++}(n)$ to the set of positive definite trace class operators on $\mathcal{H}$ of the form $A+\gamma I>0$, where $A$ is trace class, $\gamma\in\mathbb{R},\gamma>0$, and $I$ is the identity operator. This was subsequently generalized to the infinite-dimensional Alpha-Beta Log-Det divergences between positive definite trace class operators Minh:LogDet2016-AB and on the more general set of positive definite Hilbert-Schmidt operators Minh:GSI2017 . Other distance functions on the set of positive definite Hilbert-Schmidt operators include the affine-invariant Riemannian distance Larotonda:2007 Minh:GSI2015 and the Log-Hilbert-Schmidt distance MinhSB:NIPS2014 .

For a fixed $\gamma>0$, each of the above divergence/distance functions automatically becomes a divergence/distance function between covariance operators on $\mathcal{H}$. In particular, for covariance operators on reproducing kernel Hilbert spaces (RKHS), they all admit closed form expressions that can readily be employed in practical applications, see e.g. MinhSB:NIPS2014 ; Minh:CVPR2016 ; Minh:Covariance2017 . In computer vision and pattern recognition, other papers employing this approach include in ProbDistance:PAMI2006 and Covariance:CVPR2014 , in which Bregman divergences between RKHS covariance operators are applied to problems in object recognition and texture classification, among others.

It is not clear, however, how all of the above functions relate to the divergence/distance functions between Gaussian measures on the Hilbert space $\mathcal{H}$, such as the Kullback-Leibler or Rényi divergences, as is the case in the finite-dimensional setting. The aim of this work is to establish these correspondences in the case of the infinite-dimensional Alpha Log-Det divergences.

Contributions. The following are the main contributions of the current work.

We study regularized versions of the Kullback-Leibler and Rényi divergences between covariance operators and Gaussian measures on Hilbert spaces, using the infinite-dimensional Alpha Log-Det divergences. We show that for two equivalent Gaussian measures on $\mathcal{H}$, the regularized Kullback-Leibler and Rényi divergences converge to the corresponding true Kullback-Leibler and Rényi divergences, respectively, as the regularization parameter $\gamma\rightarrow 0$. 2. 2.

As part of the proof, we derive the explicit formulas for the Radon-Nikodym derivative and the true Kullback-Leibler and Rényi divergences between two equivalent Gaussian measures $\mathcal{N}(m,C)$, $\mathcal{N}(m_{0},C_{0})$ on $\mathcal{H}$, under the most general setting. These formulas generalize those available in the current literature, which assume either $C_{0}=C$ or $m_{0}=m=0$. We illustrate this with the computation of the Kullback-Leibler divergence between the posterior and prior probability measures, under the Gaussian setting, in a Bayesian inverse problem on Hilbert spaces.

Organization. The paper is structured as follows. In Section 2, we present the definitions of the regularized divergences between covariance operators and Gaussian measures on $\mathcal{H}$, using the Alpha Log-Det divergences. Section 3 summarizes the main results on the convergence of the regularized divergences to the true divergences. The proofs for the convergence are given in Sections 4 and 5. In Section 6, we present the explicit formulas for the Radon-Nikodym derivative and the true Kullback-Leibler and Rényi divergences between two equivalent Gaussian measures on $\mathcal{H}$.

Notation. Throughout the paper, we assume that $\mathcal{H}$ is a real separable Hilbert space, with $\dim(\mathcal{H})=\infty$, unless explicitly stated otherwise. Let $\mathcal{L}(\mathcal{H})$ be the Banach space of bounded linear operators on $\mathcal{H}$, with operator norm $||\;||$. Let ${\rm Sym}(\mathcal{H})\subset\mathcal{L}(\mathcal{H})$ denote the subspace of bounded, self-adjoint operators on $\mathcal{H}$. Let ${\rm Sym}^{+}(\mathcal{H})\subset{\rm Sym}(\mathcal{H})$ denote the set of self-adjoint, positive operators on $\mathcal{H}$, that is $A\in{\rm Sym}^{+}(\mathcal{H})\Longleftrightarrow\langle x,Ax\rangle\geq 0$ $\forall x\in\mathcal{H}$. Let ${\rm Sym}^{++}(\mathcal{H})\subset{\rm Sym}^{+}(\mathcal{H})$ denote the set of self-adjoint, strictly positive operators on $\mathcal{H}$, that is $\mathcal{A}\in{\rm Sym}^{++}(\mathcal{H})\Longleftrightarrow\langle x,Ax\rangle>0$ $\forall x\in\mathcal{H},x\neq 0$, or equivalently, $\ker(A)=\{0\}$.

2 Main definitions

We first present the definitions of the key concepts involved in the paper, namely the infinite-dimensional Alpha Log-Determinant divergences and the corresponding regularized divergences between Gaussian measures on Hilbert spaces. Many of these concepts were first introduced in Minh:LogDet2016 .

2.1 Infinite-dimensional Alpha Log-Det divergences between positive definite trace-class operators

In Minh:LogDet2016 , we introduced the following infinite-dimensional divergences between positive definite trace class operators on a Hilbert space $\mathcal{H}$, which generalize the Alpha Log-Determinant divergences between SPD matrices Chebbi:2012Means .

Definition 1 (Alpha Log-Determinant divergences between positive definite trace class operators)

Assume that $\dim(\mathcal{H})=\infty$. For $-1<\alpha<1$, the Log-Det $\alpha$-divergence $d^{\alpha}_{{\rm logdet}}[(A+\gamma I),(B+\mu I)]$ between $(A+\gamma I)>0,(B+\mu I)>0$, $A,B\in{\rm Tr}(\mathcal{H})$, $\gamma,\mu\in\mathbb{R}$, is defined to be

[TABLE]

where $\beta=\frac{(1-\alpha)\gamma}{(1-\alpha)\gamma+(1+\alpha)\mu}$. The limiting cases $\alpha\rightarrow\pm 1$ are defined by

[TABLE]

In Definition 1, ${\rm det_{X}}$ denotes the extended Fredholm determinant defined via ${\rm det_{X}}(A+\gamma I)=\gamma\det[(A/\gamma)+I])$, for $A\in{\rm Tr}(\mathcal{H}),\gamma\in\mathbb{R},\gamma\neq 0$, with $\det$ being the Fredholm determinant. Likewise, ${\rm tr_{X}}$ denotes the extended trace, defined by ${\rm tr_{X}}(A+\gamma I)={\rm tr}(A)+\gamma$ (see Minh:LogDet2016 for the motivations leading to these concepts).

In the case $\gamma=\mu$, $d^{\alpha}_{{\rm logdet}}[(A+\gamma I),(B+\gamma I)]$ assumes a much simpler form, which directly generalizes the finite-dimensional formulas in Chebbi:2012Means , as follows.

[TABLE]

The finite-dimensional formulas are obtained by letting $A,B\in{\rm Sym}^{++}(n)$ and $\gamma=0$.

From the above formulation, the following result is immediate.

Theorem 2.1 (Regularized divergences between covariance operators and zero-mean Gaussian measures on Hilbert spaces)

Let $-1\leq\alpha\leq 1$ be fixed. For each fixed $\gamma\in\mathbb{R}$, $\gamma>0$, the following is a divergence on the set ${\rm Sym}^{+}(\mathcal{H})\cap{\rm Tr}(\mathcal{H})$ of self-adjoint, positive trace class operators on $\mathcal{H}$

[TABLE]

Consequently, the following is a divergence on the set of Gaussian measures on $\mathcal{H}$ with mean zero and covariance operators $C_{1},C_{2}\in{\rm Sym}^{+}(\mathcal{H})\cap{\rm Tr}(\mathcal{H})$

[TABLE]

2.2 Regularized divergences between general Gaussian measures on Hilbert spaces

We next consider divergences between Gaussian measures on Hilbert spaces without the zero-mean condition. Motivated by the explicit formulas for the divergences between Gaussian densities in $\mathbb{R}^{n}$, in Minh:LogDet2016 we introduced the following regularized divergences between Gaussian measures on Hilbert spaces, using the infinite-dimensional Log-Det divergences above.

Definition 2 (Regularized Kullback-Leibler divergences between Gaussian measures on Hilbert spaces)

Let $\mathcal{N}(m_{1},C_{1})$ and $\mathcal{N}(m_{2},C_{2})$ be two Gaussian measures on $\mathcal{H}$, with corresponding mean vectors $m_{1},m_{2}\in\mathcal{H}$ and covariance operators $C_{1},C_{2}\in{\rm Sym}^{+}(\mathcal{H})\cap{\rm Tr}(\mathcal{H})$. For any fixed $\gamma\in\mathbb{R}$, $\gamma>0$, the regularized Kullback-Leibler divergence, denoted by $D^{\gamma}_{{\rm KL}}(\mathcal{N}(m_{1},C_{1})||\mathcal{N}(m_{2},C_{2}))$, is defined to be

[TABLE]

Definition 3 (Regularized Rényi divergences between Gaussian measures on Hilbert spaces)

For two Gaussian measures $\mathcal{N}(m_{1},C_{1})$ and $\mathcal{N}(m_{2},C_{2})$ on $\mathcal{H}$, the regularized Rényi divergence of order $r$, $0<r<1$, for a fixed $\gamma\in\mathbb{R}$, $\gamma>0$, denoted by $D^{\gamma}_{{\rm R},r}(\mathcal{N}(m_{1},C_{1})||\mathcal{N}(m_{2},C_{2}))$, is defined to be

[TABLE]

Remark. Our definition of the regularized Rényi divergence differs from that in Minh:LogDet2016 by a factor of $\frac{1}{r}$. It is motivated from the finite-dimensional definition $d_{R,r}(P_{1},P_{2})=-\frac{1}{r(1-r)}\log\int_{\mathbb{R}^{n}}P_{1}^{r}(x)P_{2}^{1-r}(x)dx$, see e.g. Pardo:2005 , of the Rényi divergence between two probability densities $P_{1},P_{2}$ on $\mathbb{R}^{n}$ . This differs from the original definition by Rényi Renyi:1961 , namely $d_{R,r}(P_{1},P_{2})=-\frac{1}{(1-r)}\log\int_{\mathbb{R}^{n}}P_{1}^{r}(x)P_{2}^{1-r}(x)dx$ by the factor $\frac{1}{r}$. The advantage of the current formulation is that one can see immediately that

[TABLE]

Definition 4 (Regularized Bhattacharyya and Hellinger distances between Gaussian measures on Hilbert spaces)

For two Gaussian measures $\mathcal{N}(m_{1},C_{1})$ and $\mathcal{N}(m_{2},C_{2})$ on $\mathcal{H}$, the regularized Bhattacharyya distance $D_{B}^{\gamma}(\mathcal{N}(m_{1},C_{1}),\mathcal{N}(m_{2},C_{2}))$, for a fixed $\gamma\in\mathbb{R},\gamma>0$, is defined to be

[TABLE]

The regularized Hellinger distance $D_{H}^{\gamma}(\mathcal{N}(m_{1},C_{1}),\mathcal{N}(m_{2},C_{2}))$ is defined via the regularized Bhattacharyya $D_{B}^{\gamma}(\mathcal{N}(m_{1},C_{1}),\mathcal{N}(m_{2},C_{2}))$ distance by

[TABLE]

Properties of the regularized divergences.

The regularized divergences between any pair of covariance operators, not necessarily strictly positive, are always well-defined and finite for any $\gamma>0$. Likewise, the regularized divergences between the corresponding Gaussian measures, not necessarily non-degenerate or equivalent (see below), are always well-defined and finite for any $\gamma>0$. 2. 2.

The regularized divergences between Gaussian measures are defined explicitly in terms of their mean vectors and covariance operators, not via the evaluation of the Radon-Nikodym derivatives and the corresponding integrals. 3. 3.

In the RKHS setting, when the mean vectors and covariance operators are RKHS vectors and covariance operators, respectively, all of these divergences admit closed form formulas that can be efficiently computed Minh:LogDet2016 .

3 Main theorems

The regularized divergences stated above are well-defined for any pairs of Gaussian measures on a Hilbert space $\mathcal{H}$. It is not clear from the definition, however, whether they possess a probabilistic interpretation. We now show that they are, in fact, closely related to the corresponding true divergences when the Gaussian measures under consideration are equivalent. Specifically, the following results state that, as $\gamma\rightarrow 0^{+}$, the regularized Kullback-Leibler and regularized Rényi divergences between two equivalent, non-degenerate Gaussian measures $\mathcal{N}(m_{0},C_{0})$ and $\mathcal{N}(m,C)$ converge to the true Kullback-Leibler and Rényi divergences, respectively, between $\mathcal{N}(m_{0},C_{0})$ and $\mathcal{N}(m,C)$.

Theorem 3.1 (Limiting behavior of the regularized Kullback-Leibler divergence)

*Let $\mu=\mathcal{N}(m_{0},C_{0})$ and $\nu=\mathcal{N}(m,C)$ be two non-degenerate, equivalent Gaussian measures on $\mathcal{H}$, that is with $C_{0},C\in{\rm Sym}^{++}(\mathcal{H})$. Assume that $\mu$ and $\nu$ are equivalent, that is $m-m_{0}\in{\rm Im}(C_{0}^{1/2})$ and there exists $S\in{\rm Sym}(\mathcal{H})\cap{\rm HS}(\mathcal{H})$ such that $C=C_{0}^{1/2}(I-S)C_{0}^{1/2}$. Then *

[TABLE]

where $D_{{\rm KL}}(\nu||\mu)$ denotes the Kullback-Leibler divergence between $\nu$ and $\mu$.

In Theorem 3.1, ${\rm det_{2}}$ denotes the Hilbert-Carleman determinant (see e.g. Simon:1977 ). For a Hilbert-Schmidt operator $A$, the Hilbert-Carleman determinant of $I+A$ is defined by ${\rm det_{2}}(I+A)=\det[(I+A)\exp(-A)]$. In particular, for $A\in{\rm Tr}(\mathcal{H})$, we have ${\rm det_{2}}(I+A)=\det(I+A)\exp(-{\rm tr}(A))$, and $\log{\rm det_{2}}(I+A)=\log\det(I+A)-{\rm tr}(A)$. The function ${\rm det_{2}}(I+A)$ is continuous in the Hilbert-Schmidt norm, so that $\lim_{k\rightarrow\infty}||A_{k}-A||_{{\rm HS}}=0\Rightarrow\lim_{k\rightarrow\infty}{\rm det_{2}}(I+A_{k})={\rm det_{2}}(I+A)$.

Theorem 3.1 can also be equivalently stated as

[TABLE]

where $||\;||_{C_{0}}$ is the norm corresponding to the inner product

[TABLE]

of the Cameron-Martin space $({\rm Im}(C_{0}^{1/2}),\langle\;,\;\rangle_{C_{0}})$ associated with $\mathcal{N}(m_{0},C_{0})$.

Theorem 3.2 (Limiting behavior of the regularized Rényi divergences)

Assume the hypothesis of Theorem 3.1. Let $D_{{\rm R},r}(\nu||\mu)$ denote the Rényi divergence of order $r$ between $\nu$ and $\mu$ , $0<r<1$. Then

[TABLE]

Corollary 1 (Limiting behavior of the regularized Bhattacharyya and Hellinger distances)

Assume the hypothesis of Theorem 3.1. Let $D_{B}(\nu||\mu)$ denote the true Bhattacharyya distance between $\nu$ and $\mu$. Then

[TABLE]

Similarly, let $D_{H}(\nu||\mu)$ denote the true Hellinger distance between $\nu$ and $\mu$. Then

[TABLE]

Computational consequences. The focus of the current work is on the statistical interpretation of the infinite-dimensional Alpha Log-Det divergences and the corresponding regularized divergences between Gaussian measures on Hilbert spaces. The results just stated also suggest numerical algorithms for approximating the Kullback-Leibler and Rényi divergences between probability measures on infinite-dimensional Hilbert spaces. This is an important topic, see e.g. Pinski:2015KL ,Pinski:2015KLalgorithms , which will be explored in a companion future work.

3.1 Example: KL divergences in Bayesian inverse problems on Hilbert spaces

In this section, we apply the concept of regularized KL divergences above to the setting of linear Bayesian inverse problems. As a specific example, consider the following setting from Stuart:Inverse2010 (Theorem 6.20 and Example 6.23). Let $u$ be a Gaussian random variable on the Hilbert space $\mathcal{H}$, distributed according to the Gaussian measure $\mu_{0}=\mathcal{N}(m_{0},C_{0})$, with $\ker(C_{0})=\{0\},m_{0}\in{\rm Im}(C_{0}^{1/2})$. Let $A:\mathcal{H}\rightarrow\mathbb{R}^{n}$ be a bounded linear operator. Assume that the following random variable $y\in\mathbb{R}^{n}$ is Gaussian

[TABLE]

where $\eta$ is independent of $u$. Then the random variable $y|u$ is Gaussian, with density propositional to $\exp(-\frac{1}{2}(Au-y)^{T}\Gamma^{-1}(Au-y))$. The Gaussian measure corresponding to $u|y$ is $\mu^{y}=\mathcal{N}(m,C)$, where $m$ and $C$ are given by, respectively (Stuart:Inverse2010 ),

[TABLE]

In the Bayesian setting, $\mu_{0}$ is the prior probability measure on $u$ and $\mu^{y}$ is the posterior probability measure of $u$ given the data $y$. In Alexanderian:2016 , the authors computed the KL-divergence $D_{{\rm KL}}(\mathcal{N}(m,C)||\mathcal{N}(m_{0},C_{0}))$ directly for $\Gamma=I$. We now present the general formula for $\Gamma\in{\rm Sym}^{++}(n)$, which is a straightforward consequence of the general expression for the KL-divergence given in Theorem 3.1.

Theorem 3.3

Assume that $m$ and $C$ are given by Eqs. (26) and (27), respectively. Then the KL divergence between the posterior measure $\mathcal{N}(m,C)$ and the prior measure $\mathcal{N}(m_{0},C_{0})$ is given by

[TABLE]

Special case. For $\Gamma=I$, we obtain

[TABLE]

This is precisely Eq.(19) in Proposition 3 in Alexanderian:2016 .

Remark. As noted in Alexanderian:2016 , the last term in Eq.(3.1) is precisely $\frac{1}{2}||C_{0}^{-1/2}(m-m_{0})||^{2}$. As we can see from Theorem 3.1, this term is part of the general formula for KL divergences and is not a specific feature of the Bayesian inverse problem.

4 Limiting behavior of the regularized Kullback-Leibler divergences

In this section, we prove Equation (15) in Theorem 3.1, which we restate below.

Theorem 4.1

Assume the hypothesis of Theorem 3.1. Then

[TABLE]

The first term on the right hand side of (30) follows from the following result.

Proposition 1

Assume that $\ker(C_{0})=\{0\}$. Then

[TABLE]

We first prove the following more general technical result.

Lemma 1

Let $A$ be a self-adjoint, positive, compact operator on $\mathcal{H}$. Then

[TABLE]

Assume further that $\ker(A)=\{0\}$, then for any $x\in\mathcal{H}$,

[TABLE]

Proof

Let $\{\lambda_{k}\}_{k=1}^{\infty}$ be the eigenvalues of $A$, with corresponding orthonormal eigenvectors $\{e_{k}\}_{k=1}^{\infty}$, then we have the spectral decomposition $A=\sum_{k=1}^{\infty}\lambda_{k}e_{k}\otimes e_{k}\Rightarrow A^{1/2}(A+\gamma I)^{-1}A^{1/2}=\sum_{k=1}^{\infty}\frac{\lambda_{k}}{\lambda_{k}+\gamma}e_{k}\otimes e_{k}$. For each $x\in\mathcal{H}$, write $x=\sum_{k=1}^{\infty}x_{k}e_{k}$, where $x_{k}=\langle x,e_{k}\rangle$. Then $\langle x,A^{1/2}(A+\gamma I)^{-1}A^{1/2}x\rangle=\sum_{k=1}^{\infty}\frac{\lambda_{k}}{\lambda_{k}+\gamma}x_{k}^{2}$. By Lebesgue’s Monotone Convergence Theorem, we then have $\lim_{\gamma\rightarrow 0^{+}}\langle x,A^{1/2}(A+\gamma I)^{-1}A^{1/2}x\rangle=\lim_{\gamma\rightarrow 0^{+}}\sum_{k=1}^{\infty}\frac{\lambda_{k}}{\lambda_{k}+\gamma}x_{k}^{2}=\sum_{k=1}^{\infty}\lim_{\gamma\rightarrow 0^{+}}\frac{\lambda_{k}}{\lambda_{k}+\gamma}x_{k}^{2}=\sum_{k=1}^{\infty}x_{k}^{2}=||x||^{2}.$ This proves the first identity. If $\ker(A)=\{0\}$, then we have $\lambda_{k}>0$ $\forall k\in\mathbb{N}$ and

[TABLE]

Thus for any $x\in\mathcal{H}$, we have

[TABLE]

∎

Proof (**

of Proposition 1**)

This follows from Lemma 1 by letting $x=m-m_{0}$ and $A=C_{0}$. ∎

The second term on the right hand side of (30) follows from the following result.

Assumption 1

Let $C\in{\rm Tr}(\mathcal{H}),C_{0}\in{\rm Tr}(\mathcal{H})$ be self-adjoint, positive. Assume that there exists $S\in{\rm Sym}(\mathcal{H})\cap{\rm HS}(\mathcal{H})$ such that $I-S$ is strictly positive and that

[TABLE]

Theorem 4.2

Let $C_{0},C,S$ be three bounded linear operators on $\mathcal{H}$ satisfying the hypothesis of Assumption 1. Then

[TABLE]

The right hand side is nonnegative, with zero equality if and only if $S=0$, that is if and only if $C=C_{0}$. If, in addition, $S$ is assumed to be trace class, then

[TABLE]

The limit in Theorem 4.2 follows from the continuity of the Hilbert-Carleman determinant ${\rm det_{2}}$ in the Hilbert-Schmidt norm $||\;||_{{\rm HS}}$. Its proof consists of two steps, which constitute the following two results.

Proposition 2

Let $C_{0},C$ be two self-adjoint, positive, trace class operators. Assume that there exists a self-adjoint, Hilbert-Schmidt operator $S$ such that $C=C_{0}^{1/2}(I-S)C_{0}^{1/2}$. Then for any $\gamma>0$, $\gamma\in\mathbb{R}$,

[TABLE]

Proposition 3

Let $A$ be a compact, self-adjoint, positive operator on $\mathcal{H}$. Let $B\in{\rm HS}(\mathcal{H})$. Then

[TABLE]

Lemma 2

Let $S\in{\rm Sym}(\mathcal{H})\cap{\rm HS}(\mathcal{H})$ such that $I-S$ is strictly positive. Then

[TABLE]

with equality if and only if $S=0$.

Proof

Consider the function $f(x)=\log(1-x)+x$ for $x<1$. We have $f{{}^{\prime}}(x)=-\frac{x}{1-x}$, with $f^{\prime}(x)>0$ for $x<0$ and $f^{\prime}(x)<0$ for $0<x<1$. Thus $f$ has a unique global maximum $f_{\max}=f(0)=0$. Hence $f(x)\leq 0$, with equality if and only if $x=0$.

Let $\{\lambda_{k}\}_{k=1}^{\infty}$ denote the eigenvalues of $S$, then since $I-S$ is strictly positive, we have $\lambda_{k}<1$ $\forall k\in\mathbb{N}$. Then $\log{\rm det_{2}}(I-S)=\sum_{k=1}^{\infty}[\log(1-\lambda_{k})+\lambda_{k}]\leq 0$, with equality if and only if $\lambda_{k}=0$ $\forall k\in\mathbb{N}$, that is if and only if $S=0$. ∎

Proof

(of Theorem 4.2) By Proposition 2, we have for any $\gamma>0$,

[TABLE]

By Proposition 3, we have

[TABLE]

By Theorem 6.5 in Simon:1977 , which states the continuity of the Hilbert-Carleman determinant in the Hilbert-Schmidt norm topology, we then obtain

[TABLE]

It then follows that

[TABLE]

By Lemma 2, the right hand side is always nonnegative, with zero equality if and only if $S=0$. From the expression $C=C_{0}^{1/2}(I-S)C_{0}^{1/2}$, this happens if and only if $C=C_{0}$. If $S$ is trace class, then ${\rm det_{2}}(I-S)=\det(I-S)\exp({\rm tr}(S))$ and we have

[TABLE]

∎

Proof

(of Proposition 2) By the product property of the extended Fredholm determinant (Proposition 4 in Minh:LogDet2016 ) and the commutativity of the extended trace operation (Lemma 4 in Minh:LogDet2016 ), we have

[TABLE]

For $C=C_{0}^{1/2}(I-S)C_{0}^{1/2}=C_{0}-C_{0}^{1/2}SC_{0}^{1/2}$, we have for any $\gamma>0$, $C+\gamma I=C_{0}+\gamma I-C_{0}^{1/2}SC_{0}^{1/2}$. Thus it follows that

[TABLE]

By definition of $d^{1}_{{\rm logdet}}$, we have

[TABLE]

∎

Proof of Proposition 3. We recall that a Banach space $\mathcal{B}$ is said to have the Radon-Riesz Property if $||x_{n}||\rightarrow||x||$ and $x_{n}\rightarrow x$ weakly imply that $||x_{n}-x||\rightarrow 0$ for all $\{x_{n}\}_{n\in\mathbb{N}}$ and $x$ in $\mathcal{B}$. In particular, a Hilbert space $\mathcal{H}$ possesses the Radon-Riesz Property. We now utilize this property for the Hilbert space ${\rm HS}(\mathcal{H})$, under the Hilbert-Schmidt inner product. We first prove the following.

Lemma 3

Let $A$ be a self-adjoint, positive, compact operator on $\mathcal{H}$. Then

[TABLE]

that is $(A+\gamma I)^{-1/2}A^{1/2}$ converges to $I$ in the weak operator topology as $\gamma\rightarrow 0^{+}$.

Proof

Let $\{\lambda_{k}\}_{k=1}^{\infty}$ be the eigenvalues of $A$, with corresponding orthonormal eigenvectors $\{e_{k}\}_{k=1}^{\infty}$. For any $x,y\in\mathcal{H}$, write $x=\sum_{k=1}^{\infty}x_{k}e_{k}$, $y=\sum_{k=1}^{\infty}y_{k}e_{k}$, where $x_{k}=\langle x,e_{k}\rangle$, $y_{k}=\langle y,e_{k}\rangle$. Then $\langle(A+\gamma I)^{-1/2}A^{1/2}x,y\rangle=\sum_{k=1}^{\infty}\frac{\lambda_{k}^{1/2}}{(\lambda_{k}+\gamma)^{1/2}}x_{k}y_{k}$. For each $k\in\mathbb{N}$, $\lim_{\gamma\rightarrow 0^{+}}\frac{\lambda_{k}^{1/2}}{(\lambda_{k}+\gamma)^{1/2}}x_{k}y_{k}=x_{k}y_{k}$. Furthermore,

[TABLE]

Thus by Lebesgue’s Dominated Convergence Theorem, $\lim_{\gamma\rightarrow 0^{+}}\langle(A+\gamma I)^{-1/2}A^{1/2}x,y\rangle=\lim_{\gamma\rightarrow 0^{+}}\sum_{k=1}^{\infty}\frac{\lambda_{k}^{1/2}}{(\lambda_{k}+\gamma)^{1/2}}x_{k}y_{k}=\sum_{k=1}^{\infty}\lim_{\gamma\rightarrow 0^{+}}\frac{\lambda_{k}^{1/2}}{(\lambda_{k}+\gamma)^{1/2}}x_{k}y_{k}=\sum_{k=1}^{\infty}x_{k}y_{k}=\langle x,y\rangle.$ ∎

Remark 1

Lemma 3 states that $(A+\gamma I)^{-1/2}A^{1/2}$ converges weakly to the identity operator $I$ as $\gamma\rightarrow 0^{+}$. When $\dim(\mathcal{H})=\infty$, this convergence does not hold in the operator norm topology. For any $\gamma>0$, the operator $A(A+\gamma I)^{-1}$ has eigenvalues $\{\frac{\lambda_{k}}{\lambda_{k}+\gamma}\}_{k=1}^{\infty}$, with $\lim_{\gamma\rightarrow 0^{+}}\frac{\lambda_{k}}{\lambda_{k}+\gamma}=1$. However, $\lim_{\gamma\rightarrow 0^{+}}||I-A(A+\gamma I)^{-1}||=\lim_{\gamma\rightarrow 0^{+}}\gamma||(A+\gamma I)^{-1}||\neq 0$ if $\dim(\mathcal{H})=\infty$. In fact, we have

[TABLE]

for any $\gamma>0$, since $\lim_{k\rightarrow\infty}\lambda_{k}=0$. Thus $||\gamma(A+\gamma I)^{-1}||=1$ $\forall\gamma>0$.

Lemma 4

Let $A$ be a compact, self-adjoint, positive operator on $\mathcal{H}$. Let $B\in{\rm HS}(\mathcal{H})$. Then for any operator $C\in{\rm HS}(\mathcal{H})$,

[TABLE]

i.e. $(A+\gamma I)^{-1/2}A^{1/2}BA^{1/2}(A+\gamma I)^{-1/2}$ converges weakly to $B$ in ${\rm HS}(\mathcal{H})$ as $\gamma\rightarrow 0^{+}$.

Proof

Let $\{\lambda_{k}\}_{k=1}^{\infty}$ be the eigenvalues of $A$, with corresponding orthonormal eigenvectors $\{e_{k}\}_{k=1}^{\infty}$. For any operator $C\in{\rm HS}(\mathcal{H})$, we have

[TABLE]

By Lemma 3, we have for each fixed $k\in\mathbb{N}$,

[TABLE]

Furthermore,

[TABLE]

Thus by Lebesgue’s Dominated Convergence Theorem, we then have

[TABLE]

∎

Lemma 5

Let $A$ be a compact, self-adjoint, positive operator on $\mathcal{H}$. Let $B\in{\rm HS}(\mathcal{H})$. Then

[TABLE]

Proof

Let $\{\lambda_{k}\}_{k=1}^{\infty}$ be the eigenvalues of $A$, with corresponding orthonormal eigenvectors $\{e_{k}\}_{k=1}^{\infty}$. We have for any $\gamma>0$,

[TABLE]

It follows that

[TABLE]

By Lemma 1, we have

[TABLE]

Furthermore,

[TABLE]

Thus by Lebesgue’s Dominated Convergence Theorem, we have

[TABLE]

∎

Lemma 6

Let $A\in{\rm Tr}(\mathcal{H})$ be self-adjoint, positive. Let $B\in\mathcal{L}(\mathcal{H})$. Then

[TABLE]

If $B\in{\rm HS}(\mathcal{H})$, then

[TABLE]

Proof

Since $A$ and $(A+\gamma I)$ commute, we have

[TABLE]

Since $A$ is trace class, self-adjoint, positive, $A^{1/2}\in{\rm HS}(\mathcal{H})$, so that for $B\in\mathcal{L}(\mathcal{H})$, $A^{1/2}B\in{\rm HS}(\mathcal{H})$, $BA^{1/2}\in{\rm HS}(\mathcal{H})$. We then have

[TABLE]

If $B\in{\rm HS}(\mathcal{H})$, then we have $A^{1/2}B\in{\rm Tr}(\mathcal{H}),BA^{1/2}\in{\rm Tr}(\mathcal{H})$ and

[TABLE]

∎

Proof

(of Proposition 3) By Lemma 4, we have for any $C\in{\rm HS}(\mathcal{H})$,

[TABLE]

that is the operator $(A+\gamma I)^{-1/2}A^{1/2}BA^{1/2}(A+\gamma I)^{-1/2}$ converges weakly to $B$ on ${\rm HS}(\mathcal{H})$ as $\gamma\rightarrow 0^{+}$. By Lemma 5,

[TABLE]

Thus Radon-Riesz Property can be invoked to give

[TABLE]

∎

Proof (**

of Theorem 4.1**)

This follows from Proposition 1 and Theorem 4.2. ∎

5 Limiting behavior of the

regularized Rényi divergence

In this section, we prove Equation(19) in Theorem 3.2, which we restate below.

Theorem 5.1

Assume the hypothesis of Theorem 3.2. Then

[TABLE]

We need the following technical results.

Lemma 7 (Minh:LogDet2016-AB )

Let $0<r\leq 1$ be fixed. Let $\{A_{n}\}_{n\in\mathbb{N}}\in{\rm Sym}(\mathcal{H})\cap{\rm HS}(\mathcal{H})$, $A\in{\rm Sym}(\mathcal{H})\cap{\rm HS}(\mathcal{H})$ be such that $I+A>0$, $I+A_{n}>0$ $\forall n\in\mathbb{N}$. Assume that $\lim_{n\rightarrow\infty}||A_{n}-A||_{{\rm HS}}=0$. Then

[TABLE]

Proposition 4

Let $0<r<1$ be fixed. For $m,m_{0}\in\mathcal{H}$ and two self-adjoint, compact, positive operators $C,C_{0}$ on $\mathcal{H}$,

[TABLE]

In particular, $||[(1-r)C+rC_{0}]^{-1/2}(m-m_{0})||^{2}<\infty$ for $m-m_{0}\in{\rm Im}(C_{0}^{1/2})$.

Proof

By Lemma 1,

[TABLE]

By Theorem 2.2 in Fillmore:1971Operator , for any two bounded operators $A,B$ on $\mathcal{H}$,

[TABLE]

In particular, for any two self-adjoint, positive bounded operators $A,B$ on $\mathcal{H}$,

[TABLE]

Since $0\in{\rm Im}(A^{1/2})$, $0\in{\rm Im}(B^{1/2})$, this implies that ${\rm Im}(A^{1/2})\subset{\rm Im}[(A+B)^{1/2}]$, ${\rm Im}(B^{1/2})\subset{\rm Im}[(A+B)^{1/2}]$, and we have

[TABLE]

Thus if $m-m_{0}\in{\rm Im}(C_{0}^{1/2})$, then $m-m_{0}\in{\rm Im}[(1-r)C+rC_{0}]^{1/2}$ for $0<r<1$ and

[TABLE]

∎

Proof (**

of Theorem 5.1**)

By definition of the regularized Renyi divergence, Eq.(3),

[TABLE]

For the first term, we have

[TABLE]

Thus by Proposition 4, we have for $m-m_{0}\in{\rm Im}(C_{0}^{1/2})$,

[TABLE]

Let $\{\beta_{k}\}_{k\in\mathbb{N}}$ be the eigenvalues of $C_{0}^{1/2}(I-(1-r)S)C_{0}^{1/2}$, with corresponding orthonormal eigenvectors $\{\varphi_{k}\}_{k\in\mathbb{N}}$. Since $\ker(C_{0})=\{0\}$, we have $\beta_{k}>0$ $\forall k\in\mathbb{N}$. Then $\{\frac{(I-(1-r)S)^{1/2}C_{0}^{1/2}\varphi_{k}}{\sqrt{\beta_{k}}}\}_{k\in\mathbb{N}}$ are the orthonormal eigenvectors of $(I-(1-r)S)^{1/2}C_{0}(I-(1-r)S)^{1/2}$, with the same eigenvalues. Thus

[TABLE]

For the second term, by Definition 1,

[TABLE]

For $C=C_{0}^{1/2}(I-S)C_{0}^{1/2}=C_{0}-C_{0}^{1/2}SC_{0}^{1/2}$, we have

[TABLE]

Thus the extended Fredholm determinant of $(C_{0}+\gamma I)^{-1/2}(C+\gamma I)(C_{0}+\gamma I)^{-1/2}$ is the Fredholm determinant of $I-(C_{0}+\gamma I)^{-1/2}C_{0}^{1/2}SC_{0}^{1/2}(C_{0}+\gamma I)^{-1/2}$ and consequently

[TABLE]

where $A_{S,\gamma}=(C_{0}+\gamma I)^{-1/2}C_{0}^{1/2}SC_{0}^{1/2}(C_{0}+\gamma I)^{-1/2}$.

By Proposition 3, we have $\lim_{\gamma\rightarrow 0^{+}}||A_{S,\gamma}-S||_{{\rm HS}}=0$. By Lemma 7,

[TABLE]

We then exploit the property that $||A_{1}A_{2}||_{{\rm tr}}\leq||A_{1}||_{{\rm HS}}||A_{2}||_{{\rm HS}}$ for any two Hilbert-Schmidt operators $A_{1},A_{2}$ (see e.g. ReedSimon:Functional ). This gives us

[TABLE]

By the continuity of the Fredholm determinant with respect to the trace norm (see e.g. Theorem 3.5 in Simon:1977 ), we then obtain

[TABLE]

∎

6 The Radon-Nikodym derivatives and divergences between Gaussian measures on Hilbert spaces

For completeness, we now derive the explicit formulas for the exact Kullback-Leibler and Rényi divergences between two equivalent Gaussian measures, that is Eq. (16) in Theorem 3.1 and Eq. (20) in Theorem 3.2.

Throughout the following, we utilize the white noise mapping, see e.g. DaPrato:2006 ; DaPrato:PDEHilbert . Let $m\in\mathcal{H}$ and $Q$ be a self-adjoint, positive trace class operator on $\mathcal{H}$. Assume that $\ker(Q)=\{0\}$, then the Gaussian measure $\mu=\mathcal{N}(m,Q)$ is said to be non-degenerate. Let $\{\lambda_{k}\}_{k=1}^{\infty}$ be the eigenvalues of $Q$, with corresponding orthonormal eigenvectors $\{e_{k}\}_{k=1}^{\infty}$, then $\lambda_{k}>0$ $\forall k\in\mathbb{N}$, with $\lim_{k\rightarrow\infty}\lambda_{k}=0$. The inverse operator $Q^{-1}:{\rm Im}(Q)\rightarrow\mathcal{H}$ is unbounded, since $Q^{-1}e_{k}=\frac{1}{\lambda_{k}}e_{k}$ with $||Q^{-1}e_{k}||=\frac{1}{\lambda_{k}}\rightarrow\infty$ as $k\rightarrow\infty$. For $r\geq 0$, define the following subspace

[TABLE]

For $r=\frac{1}{2}$, the space $Q^{1/2}(\mathcal{H})={\rm Im}(Q^{1/2})$ is called the Cameron-Martin space associated with the Gaussian measure $\mathcal{N}(m,Q)$. It is a Hilbert space with inner product

[TABLE]

In the following, for $\mu=\mathcal{N}(m,Q)$, we define

[TABLE]

White noise mapping. Consider the following mapping

[TABLE]

For any pair $z_{1},z_{2}\in Q^{1/2}(\mathcal{H})$, we have by definition of the covariance operator

[TABLE]

Thus the map $W:Q^{1/2}(\mathcal{H})\rightarrow\mathcal{L}^{2}(\mathcal{H},\mu)$ is an isometry, that is

[TABLE]

Since $\ker(Q)=\{0\}$, the subspace $Q^{1/2}(\mathcal{H})$ is dense in $\mathcal{H}$ and the map $W$ can be uniquely extended to all of $\mathcal{H}$, as follows. For any $z\in\mathcal{H}$, let $\{z_{n}\}_{n\in\mathbb{N}}$ be a sequence in $Q^{1/2}(\mathcal{H})$ with $\lim_{n\rightarrow\infty}||z_{n}-z||_{\mathcal{H}}=0$. Then $\{z_{n}\}_{n\in\mathbb{N}}$ is a Cauchy sequence in $\mathcal{H}$, so that by isometry, $\{W_{z}\}_{n\in\mathbb{N}}$ is also a Cauchy sequence in $\mathcal{L}^{2}(\mathcal{H},\mu)$, thus converging to a unique element in $\mathcal{L}^{2}(\mathcal{H},\mu)$. Thus for any $z\in\mathcal{H}$, we can define the map

[TABLE]

by the following unique limit in $\mathcal{L}^{2}(\mathcal{H},\mu)$

[TABLE]

The map $W:\mathcal{H}\rightarrow\mathcal{L}^{2}(\mathcal{H},\mu)$ is called the white noise mapping associated with the measure $\mu=\mathcal{N}(m,Q)$. One sees immediately that $W$ maps any orthonormal sequence $\{\phi_{k}\}_{k\in\mathbb{N}}$ in $\mathcal{H}$ to an orthonormal sequence $\{W_{\phi_{k}}\}_{k=1}^{\infty}$ in $\mathcal{L}^{2}(\mathcal{H},\mu)$, since

[TABLE]

Furthermore, the random variables $\{\mathcal{W}_{\phi_{k}}\}_{k=1}^{N}$ are independent (DaPrato:2006 , Proposition 1.28).

White noise mapping via finite-rank orthogonal projections. $W_{z}$ can be expressed explicitly in terms of the finite-rank orthogonal projections $P_{N}=\sum_{k=1}^{N}e_{k}\otimes e_{k}$ onto the $N$-dimensional subspaces of $\mathcal{H}$ spanned by $\{e_{k}\}_{k=1}^{N}$, $N\in\mathbb{N}$, where $\{e_{k}\}_{k\in\mathbb{N}}$ are the orthonormal eigenvectors of $Q$. For any $z\in\mathcal{H}$, we have

[TABLE]

Thus $Q^{-1/2}P_{N}z$ is always well-defined $\forall z\in\mathcal{H}$. Furthermore, for all $x,y\in\mathcal{H}$,

[TABLE]

In other words, the operator $Q^{-1/2}P_{N}:\mathcal{H}\rightarrow\mathcal{H}$ is bounded and self-adjoint $\forall N\in\mathbb{N}$. Since the sequence $\{P_{N}z\}_{N\in\mathbb{N}}$ converges to $z$ in $\mathcal{H}$, we have, in the $\mathcal{L}^{2}(\mathcal{H},\mu)$ sense,

[TABLE]

The Radon-Nikodym derivatives between Gaussian measures. Given their importance, these objects have been studied extensively, e.g. Capon:Radon1964 ; Shepp:1966Radon ; Henrich:Gaussian1972 ; DaPrato:2006 ; DaPrato:PDEHilbert ; Bogachev:Gaussian . However, the explicit formulas available in the literature generally consider two separate cases, namely two Gaussian measures both with mean zero or with the same covariance operator. We now present an explicit formula for the general case.

In the following, let $Q,R$ be two self-adjoint, positive trace class operators on $\mathcal{H}$ such that $\ker(Q)=\ker(R)=\{0\}$. Let $m_{1},m_{2}\in\mathcal{H}$. A fundamental result in the theory of Gaussian measures is the Feldman-Hajek Theorem Feldman:Gaussian1958 , Hajek:Gaussian1958 , which states that two Gaussian measures $\mu=\mathcal{N}(m_{1},Q)$ and $\nu=\mathcal{N}(m_{2},R)$ are either mutually singular or mutually equivalent. The necessary and sufficient conditions for the equivalence of the two Gaussian measures $\nu$ and $\mu$ are given by the following.

Theorem 6.1 (Bogachev:Gaussian , Corollary 6.4.11, DaPrato:PDEHilbert , Theorems 1.3.9 and 1.3.10)

*Let $\mathcal{H}$ be a separable Hilbert space. Consider two Gaussian measures $\mu=\mathcal{N}(m_{1},Q)$ and $\nu=\mathcal{N}(m_{2},R)$ on $\mathcal{H}$. Then $\mu$ and $\nu$ are equivalent if and only if the following hold *

$m_{2}-m_{1}\in{\rm Im}(Q^{1/2})$. 2. 2.

There exists $S\in{\rm Sym}(\mathcal{H})\cap{\rm HS}(\mathcal{H})$, without the eigenvalue $1$, such that

[TABLE]

For any $A\in\mathcal{L}(\mathcal{H})$, we have ${\rm Im}(A)={\rm Im}((AA^{*})^{1/2})$ Fillmore:1971Operator , thus Eq.(66) implies

[TABLE]

We assume from now on that $\mu$ and $\nu$ are equivalent. In Corollary 6.4.11 in Bogachev:Gaussian , an explicit formula for the Radon-Nikodym derivative $\frac{d\nu}{d\mu}$ is given when $m_{1}=m_{2}=0$. In Proposition 1.3.11 in DaPrato:PDEHilbert , an explicit formula is given when $m_{1}=m_{2}=0$ and $S$ is trace class. In the following, we present an explicit formula for the general case.

Let $\{\alpha_{k}\}_{k\in\mathbb{N}}$ be the eigenvalues of $S$, with corresponding orthonormal eigenvectors $\{\phi_{k}\}_{k\in\mathbb{N}}$, which form an orthonormal basis in $\mathcal{H}$. The following result expresses the Radon-Nikodym derivative $\frac{d\nu}{d\mu}$ in terms of the $\alpha_{k}$’s and $\phi_{k}$’s.

Theorem 6.2

Let $\mu=\mathcal{N}(m_{1},Q)$, $\nu=\mathcal{N}(m_{2},R)$, with $m_{2}-m_{1}\in{\rm Im}(Q^{1/2})$, $R=Q^{1/2}(I-S)Q^{1/2}$. The Radon-Nikodym derivative $\frac{d\nu}{d\mu}$ is given by

[TABLE]

where for each $k\in\mathbb{N}$

[TABLE]

The series $\sum_{k=1}^{\infty}\Phi_{k}$ converges in $\mathcal{L}^{1}(\mathcal{H},\mu)$ and $\mathcal{L}^{2}(\mathcal{H},\mu)$ and the function $s(x)=\exp\left[-\frac{1}{2}\sum_{k=1}^{\infty}\Phi_{k}(x)\right]\in\mathcal{L}^{1}(\mathcal{H},\mu)$.

Special case. For $m_{1}=m_{2}=0$, Theorem 6.2 gives

[TABLE]

This is essentially Eq. (6.4.13) in Corollary 6.4.11 in Bogachev:Gaussian .

Corollary 2

Assume the hypothesis of Theorem 6.2. Assume further that $S$ is trace class. The Radon-Nikodym derivative of $\nu$ with respect to $\mu$ is given by

[TABLE]

In the above expression,

[TABLE]

with the limits being in the $\mathcal{L}^{1}(\mathcal{H},\mu)$ and $\mathcal{L}^{2}(\mathcal{H},\mu)$ sense, respectively.

Special case. For $m_{1}=m_{2}=0$ and $S$ trace class, Corollary 2 gives

[TABLE]

This is precisely Proposition 1.3.11 in DaPrato:PDEHilbert .

Special case. If $Q=R$, then obviously $S=0$ and Corollary 2 gives

[TABLE]

This is precisely Theorem 6.14 in Stuart:Inverse2010 .

Special case: Radon-Nikodym derivative between Gaussian densities on $\mathbb{R}^{n}$. Let $P_{1}\sim\mathcal{N}(\mu_{1},\Sigma_{1})$, $P_{2}\sim\mathcal{N}(\mu_{2},\Sigma_{2})$, with $\mu_{1},\mu_{2}\in\mathbb{R}^{n}$, $\Sigma_{1},\Sigma_{2}\in{\rm Sym}^{++}(n)$. Let $S\in{\rm Sym}(n)$ be such that $\Sigma_{2}=\Sigma_{1}^{1/2}(I-S)\Sigma_{1}^{1/2}$, then one can verify directly that

[TABLE]

To prove Theorem 6.2, we first prove the following.

Proposition 5

Assume that $\ker(Q)=\ker(R)=\{0\}$ and that $R=Q^{1/2}(I-S)Q^{1/2}$, where $S\in{\rm Sym}(\mathcal{H})\cap{\rm HS}(\mathcal{H})$. Then the operator $(I-S)$ is necessarily strictly positive, that is $\langle x,(I-S)x\rangle>0$ $\forall 0\neq x\in\mathcal{H}$.

Proof

For any $x\in\mathcal{H}$, we have

[TABLE]

with equality if and only if $x=0$, since $\ker(R)=\{0\}$. Thus we have

[TABLE]

with equality if and only if $y=0$. Since $\ker(Q)=\{0\}$, ${\rm Im}(Q^{1/2})$ is dense in $\mathcal{H}$ and $\forall y\in\mathcal{H}$, $\exists$ a sequence $\{y_{n}\}_{n\in\mathbb{N}}$ in ${\rm Im}(Q^{1/2})$ such that $\lim_{n\rightarrow\infty}||y_{n}-y||=0$. One has

[TABLE]

It follows that $\langle y,(I-S)y\rangle=\lim_{n\rightarrow\infty}\langle y_{n},(I-S)y_{n}\rangle\geq 0$. Hence the operator $I-S$ is self-adjoint, positive on $\mathcal{H}$.

Let us show that $I-S$ is strictly positive. Assume that $\exists y\neq 0\in\mathcal{H}$ such that $\langle y,(I-S)y\rangle=0$, then $y\notin{\rm Im}(Q^{1/2})$ and there exists a sequence $\{y_{n}\}_{n\in\mathbb{N}}$ in ${\rm Im}(Q^{1/2})$ such that $\lim_{n\rightarrow\infty}||y_{n}-y||=0$ and $\lim_{n\rightarrow\infty}\langle y_{n},(I-S)y_{n}\rangle=0$. Equivalently, there exists a sequence $\{x_{n}\}_{n\in\mathbb{N}}$ in $\mathcal{H}$ such that $y_{n}=Q^{1/2}x_{n}$ and

[TABLE]

This implies that for any $z\in\mathcal{H}$, we have

[TABLE]

Since $\ker(R)=\{0\}$, ${\rm Im}(R^{1/2})$ is dense in $\mathcal{H}$ and thus $\lim_{n\rightarrow\infty}\langle x_{n},z\rangle=0\;\;\forall z\in\mathcal{H}$. Thus the sequence $\{x_{n}\}_{n\in\mathbb{N}}$ converges weakly to zero in $\mathcal{H}$. Then for any $z\in\mathcal{H}$,

[TABLE]

Thus the sequence $\{y_{n}\}_{n\in\mathbb{N}}$ also converges weakly to zero in $\mathcal{H}$. Since we already assume that $y_{n}$ converges strongly, and hence weakly, to $y\in\mathcal{H}$, by the uniqueness of the weak limit, we must have $y=0$, contradicting our prior assumption that $y\neq 0$. ∎

In the following, we make use of the Vitali Convergence Theorem (see e.g. Folland:Real ; Rudin:RealComplex ). Let $(\mathcal{X},\mathcal{F},\mu)$ be a positive measurable space. A sequence of functions $\{f_{n}\}_{n\in\mathbb{N}}\in\mathcal{L}^{1}(\mathcal{X},\mu)$ is said to be uniformly integrable if $\forall\epsilon>0$ $\exists\delta>0$ such that

[TABLE]

Theorem 6.3 (Vitali Convergence Theorem)

Assume that $(\mathcal{X},\mathcal{F},\mu)$ is a positive measurable space with $\mu(\mathcal{X})<\infty$. Let $\{f_{n}\}_{n\in\mathbb{N}}$ be a sequence of functions that are uniformly integrable on $\mathcal{X}$, with $f_{n}\rightarrow f$ a.e. and $|f|<\infty$ a.e.. Then $f\in\mathcal{L}^{1}(\mathcal{X},\mu)$ and $||f_{n}-f||_{\mathcal{L}^{1}(\mathcal{X},\mu)}\rightarrow 0$.

Proposition 6

Let $g\in\mathcal{H}$. Let $c_{1}\in\mathbb{R}$, $c_{2}\in\mathbb{R}$ be such that $c_{1}||g||^{2}<1$. Then

[TABLE]

Special case. For $c_{1}=0$, Proposition 6 gives

[TABLE]

With $c_{2}=1$, the above formula gives Proposition 1.2.7 in DaPrato:PDEHilbert .

The proof of Proposition 6 requires the following results. The first one, Lemma 8, can be directly verified.

Lemma 8

Let $u\in\mathcal{H}$ and $c\in\mathbb{R}$ be such that $c||u||^{2}<1$. Then the operator $I-c(u\otimes u)$ is invertible and

[TABLE]

In particular, $[I-c(u\otimes u)]^{-1}u=\frac{1}{1-c||u||^{2}}u$.

The second is the following result from DaPrato:PDEHilbert .

Theorem 6.4 (DaPrato:PDEHilbert , Proposition 1.2.8)

Assume that $M$ is a self-adjoint operator on $\mathcal{H}$ such that $\langle Q^{1/2}MQ^{1/2}x,x\rangle<||x||^{2}$ $\forall x\in\mathcal{H},x\neq 0$. Let $b\in\mathcal{H}$. Then

[TABLE]

Proof

(of Proposition 6) It suffices to prove for $m=0$. We apply Theorem 6.4 as follows. Let $\{P_{N}\}_{N\in\mathbb{N}}$, $P_{N}=\sum_{j=1}^{N}e_{j}\otimes e_{j}$ be the sequence of orthogonal projections in $\mathcal{H}$ corresponding to the eigenvectors $\{e_{j}\}_{j\in\mathbb{N}}$ of $Q$. Consider the limit

[TABLE]

Let $N\in\mathbb{N}$ be fixed. We have

[TABLE]

Let $M=c_{1}[(Q^{-1/2}P_{N}g)\otimes(Q^{-1/2}P_{N}g)]$, $b=c_{2}(Q^{-1/2}P_{N}g)$. Then for any $x\in\mathcal{H}$,

[TABLE]

which implies that $Q^{1/2}MQ^{1/2}=c_{1}(P_{N}g)\otimes(P_{N}g)$, which is a rank-one operator with eigenvalue $c_{1}||P_{N}g||^{2}$. If $c_{1}<0$, then obviously $c_{1}||P_{N}g||^{2}<1$. If $c_{1}\geq 0$, then $c_{1}||P_{N}g||^{2}\leq c_{1}||g||^{2}<1$. Also, $Q^{1/2}b=c_{2}P_{N}g$. By Lemma 8, the operator $(I-Q^{1/2}MQ^{1/2})$ is invertible, with

[TABLE]

It follows that

[TABLE]

By the assumption that $c_{1}||g||^{2}<1$, there exists $p>1$ such that $pc_{1}||g||^{2}<1$, so that $pc_{1}||P_{N}g||^{2}<1$ $\forall N\in\mathbb{N}$. Hence by Theorem 6.4, we have

[TABLE]

Taking limit as $N\rightarrow\infty$ gives

[TABLE]

Hence it follows, by applying from Hölder’s Inequality, that the sequence of functions $\left\{\exp\left[\frac{1}{2}c_{1}W_{P_{N}g}^{2}(x)+c_{2}W_{P_{N}g}(x)\right]\right\}_{N\in\mathbb{N}}$ is uniformly integrable. Thus we can apply Vitali’s Convergence Theorem to obtain

[TABLE]

∎

Proposition 7

Assume the hypothesis of Theorem 6.2. There exists $p>1$ such that $I+(p-1)S>0$. Define $s(x)=\exp\left[-\frac{1}{2}\sum_{k=1}^{\infty}\Phi_{k}(x)\right]$, where $\Phi_{k}$ is defined by Eq. (69) in Theorem 6.2. Then $s\in\mathcal{L}^{q}(\mathcal{H},\mu)$ for all $q$ satisfying $0<q<p$, with

[TABLE]

In particular, for $q=1$,

[TABLE]

Furthermore, for $s_{N}(x)=\exp\left[-\frac{1}{2}\sum_{k=1}^{N}\Phi_{k}(x)\right]$, the sequence $\{s_{N}^{q}\}_{N\in\mathbb{N}}$ is uniformly integrable on $(\mathscr{B}(\mathcal{H}),\mu)$ for $0<q<p$.

Proof

For each fixed $k\in\mathbb{N}$, we recall that the function $\Phi_{k}$ is given by

[TABLE]

We first claim that there exists $p>1$ such that $1+(p-1)\alpha_{k}>0$ $\forall k\in\mathbb{N}$. Since $\lim_{k\rightarrow\infty}\alpha_{k}=0$, there exists $\mu>0$ such that $\alpha_{k}\geq-\mu$ $\forall k\in\mathbb{N}$. Let $p$ be such that $1<p<\frac{1}{\mu}+1$, so that $(p-1)\mu<1$. Then

[TABLE]

Similarly, we have $I+(q-1)S>0$ for all $q$ satisfying $1\leq q<p$. Recall that since $I-S>0$, we have $\alpha_{k}<1$ $\forall k\in\mathbb{N}$. For $q$ satisfying $0<q<1$, we have $1-(1-q)\alpha_{k}>0$ when $\alpha_{k}<0$ and $1-(1-q)\alpha_{k}\geq 1-\alpha_{k}>0$ for $0\leq\alpha_{k}<1$. It follows that $I+(q-1)S>0$ for all $q$ satisfying $0<q<1$. Hence

[TABLE]

For each $k\in\mathbb{N}$, by Proposition 6, with $c_{1}=-\frac{p\alpha_{k}}{1-\alpha_{k}},c_{2}=\frac{p}{1-\alpha_{k}}\langle Q^{-1/2}(m_{2}-m_{1}),\phi_{k}\rangle$,

[TABLE]

For each $N\in\mathbb{N}$, consider the nonnegative function $s_{N}(x)=\exp\left[-\frac{1}{2}\sum_{k=1}^{N}\Phi_{k}(x)\right]$. By the independence of the functions $W_{\phi_{k}}$, we have

[TABLE]

Since $-1/(p-1)<\alpha_{k}<1$ $\forall k\in\mathbb{N}$, by Lemma 22 we have

[TABLE]

Since $\sum_{k=1}^{\infty}\alpha_{k}^{2}<\infty$, $\exists$ $N_{0}\in\mathbb{N}$ such that $|\alpha_{k}|<1/2$ $\forall k\geq N_{0}$. Then by Lemma 22,

[TABLE]

Thus it follows that

[TABLE]

It follows that the sequence $\left\{\exp\left(-\frac{1}{2}\sum_{k=1}^{N}[(p-1)\log(1-\alpha_{k})+\log(1+(p-1)\alpha_{k})]\right)\right\}_{N\in\mathbb{N}}$ is increasing towards the limit $\exp\left(-\frac{1}{2}\sum_{k=1}^{\infty}[(p-1)\log(1-\alpha_{k})+\log(1+(p-1)\alpha_{k})]\right)$. Hence the sequence $\{\int_{\mathcal{H}}s_{N}^{p}(x)d\mu(x)\}_{N\in\mathbb{N}}$ is increasing towards the limit

[TABLE]

By Hölder’s Inequality, for any $0<q<p$, for any set $A\in\mathscr{B}(\mathcal{H})$, we have

[TABLE]

Combining with the limit for $\{\int_{\mathcal{H}}s_{N}^{p}(x)d\mu(x)\}_{N\in\mathbb{N}}$, this shows that the sequence $\{s_{N}^{q}(x)\}$ is uniformly integrable on $(\mathscr{B}(\mathcal{H}),\mu)$. By Vitali’s Convergence Theorem,

[TABLE]

Thus it follows that $s(x)=\exp\left(-\frac{1}{2}\sum_{k=1}^{\infty}\Phi_{k}(x)\right)\in\mathcal{L}^{q}(\mathcal{H},\mu)$. In particular, for $q=1$,

[TABLE]

∎

Lemma 9

For any $a\in\mathcal{H}$, we have $W_{a}^{2}\in\mathcal{L}^{2}(\mathcal{H},\mu)$. For any $a,b\in\mathcal{H}$,

[TABLE]

In particular, for $a=b$, $\int_{\mathcal{H}}W_{a}^{4}(x)\mathcal{N}(m,Q)(dx)=3||a||^{4}$. For any two $a,b\in\mathcal{H}$,

[TABLE]

In particular, an orthonormal sequence $\{a_{k}\}_{k\in\mathbb{N}}$ in $\mathcal{H}$ gives rise to an orthonormal sequence $\{\frac{1}{\sqrt{2}}(W_{a_{k}}^{2}-1)\}_{k\in\mathbb{N}}$ in $\mathcal{L}^{2}(\mathcal{H},\mu)$ (see also DaPrato:PDEHilbert , Proposition 1.2.6).

Proof

For $a,b\in Q^{1/2}(\mathcal{H})$, by Lemma 19, we have

[TABLE]

Let $a\in\mathcal{H}$. Since $Q^{1/2}(\mathcal{H})$ is dense in $\mathcal{H}$, let $\{a_{n}\}_{n\in\mathbb{N}}$ be a Cauchy sequence in $\mathcal{H}$ with $a_{n}\in Q^{1/2}(\mathcal{H})$ and $\lim_{n\rightarrow\infty}||a_{n}-a||=0$. Then $W_{a_{n}}\rightarrow W_{a}$ in $\mathcal{L}^{2}(\mathcal{H},\mu)$. The previous identity gives

[TABLE]

The hypothesis $\lim_{n,m\rightarrow\infty}||a_{n}-a_{m}||=0$ and the above identity show that $\lim_{n,m\rightarrow\infty}||W_{a_{n}}^{2}-W_{a_{m}}^{2}||_{\mathcal{L}^{2}(\mathcal{H},\mu)}=0$. Thus $\{W_{a_{n}}^{2}\}_{n\in\mathbb{N}}$ is a Cauchy sequence in $\mathcal{L}^{2}(\mathcal{H},\mu)$ and hence converges to a unique element in $\mathcal{L}^{2}(\mathcal{H},\mu)$, which must be $W_{a}^{2}$. Thus $W_{a}^{2}\in\mathcal{L}^{2}(\mathcal{H},\mu)$.

Let $b\in\mathcal{H}$ with the corresponding Cauchy sequence $\{b_{n}\}_{n\in\mathbb{N}}$, $b_{n}\in Q^{1/2}(\mathcal{H})$. Then

[TABLE]

This give us the first and second identities. The third identity follows from the first by invoking the isometry $||W_{a}||^{2}_{\mathcal{L}^{2}(\mathcal{H},\mu)}=||a||^{2}\;\;\;\forall a\in\mathcal{H}$. ∎

Lemma 10

Consider the functions

[TABLE]

Then $\lim_{N\rightarrow\infty}||f_{N}-f||_{\mathcal{L}^{2}(\mathcal{H},\mu)}=0,\;\;\;\lim_{N\rightarrow\infty}||f_{N}-f||_{\mathcal{L}^{1}(\mathcal{H},\mu)}=0$.

Proof

By Lemma 9, the functions $\{\frac{1}{\sqrt{2}}(W_{\phi_{k}}^{2}-1)\}_{k\in\mathbb{N}}$ are orthonormal in $\mathcal{L}^{2}(\mathcal{H},\mu)$. We rewrite $f_{N}$ as

[TABLE]

Consider the functions

[TABLE]

Since $\sum_{k=1}^{\infty}\alpha_{k}^{2}<\infty$, there exists $N_{0}\in\mathbb{N}$ such that $|\alpha_{k}|<1/2$ $\forall k>N_{0}$. By Lemma 9, we have for all $N\geq N_{0}$,

[TABLE]

Consider next the series

[TABLE]

By Lemma 21, we have , since $\alpha_{k}<1$ $\forall k\in\mathbb{N}$,

[TABLE]

It thus follows that for al $N\geq N_{0}$,

[TABLE]

as $N\rightarrow\infty$. Thus the series $\sum_{k=1}^{\infty}\left[\frac{\alpha_{k}}{1-\alpha_{k}}+\log(1-\alpha_{k})\right]$ converges to a finite positive value. Together with $\lim_{N\rightarrow\infty}||h_{N}-h||_{\mathcal{L}^{2}(\mathcal{H},\mu)}=0$, this implies that $\lim_{N\rightarrow\infty}||f_{N}-f||_{\mathcal{L}^{2}(\mathcal{H},\mu)}=0$. Since $\mu$ is a probability measure, by Hölder’s Inequality, we have $||f_{N}-f||_{\mathcal{L}^{1}(\mathcal{H},\mu)}\leq\sqrt{\mu(\mathcal{H})}||f_{N}-f||_{\mathcal{L}^{2}(\mathcal{H},\mu)}=||f_{N}-f||_{\mathcal{L}^{2}(\mathcal{H},\mu)}\rightarrow 0$ as $N\rightarrow\infty$. ∎

Lemma 11

Consider the functions

[TABLE]

Then $g\in\mathcal{L}^{2}(\mathcal{H},\mu)$, $g\in\mathcal{L}^{1}(\mathcal{H},\mu)$, and

[TABLE]

Proof

Since the functions $\{W_{\phi_{k}}\}_{k\in\mathbb{N}}$ are orthonormal in $\mathcal{L}^{2}(\mathcal{H},\mu)$, we have

[TABLE]

Thus $g\in\mathcal{L}^{2}(\mathcal{H},\mu)$ and

[TABLE]

Since $\mu$ is a probability measure, by Hölder’s Inequality, we have $||g_{N}-g||_{\mathcal{L}^{1}(\mathcal{H},\mu)}\leq\sqrt{\mu(\mathcal{H})}||g_{N}-g||_{\mathcal{L}^{2}(\mathcal{H},\mu)}=||g_{N}-g||_{\mathcal{L}^{2}(\mathcal{H},\mu)}\rightarrow 0$ as $N\rightarrow\infty$. ∎

The following is a direct generalization of Claim 1 in Proposition 1.2.8 in DaPrato:PDEHilbert .

Lemma 12

Let $\{\phi_{k}\}_{k=1}^{\infty}$ be any orthonormal basis in $\mathcal{H}$. For any $b\in\mathcal{H}$,

[TABLE]

where the series converges in $\mathcal{L}^{2}(\mathcal{H},\mathcal{N}(m,Q))$.

Proof (**

of Theorem 6.2**)

By Lemmas 10 and 11, the series $\sum_{k=1}^{\infty}\Phi_{k}$ converges in $\mathcal{L}^{1}(\mathcal{H},\mu)$ and $\mathcal{L}^{2}(\mathcal{H},\mu)$. By Proposition 7, $s(x)=\exp\left[-\frac{1}{2}\sum_{k=1}^{\infty}\Phi_{k}(x)\right]\in\mathcal{L}^{1}(\mathcal{H},\mu)$, with $\int_{\mathcal{H}}s(x)d\mu(x)=\exp\left[\frac{1}{2}||(I-S)^{-1/2}Q^{-1/2}(m_{2}-m_{1})||^{2}\right]$. Define

[TABLE]

Then $\rho$ is nonnegative and satisfies $\rho\in\mathcal{L}^{1}(\mathcal{H},\mu)$, with $\int_{\mathcal{H}}\rho(x)d\mu(x)=1$, i.e. $\rho\mu$ is a probability measure on $\mathscr{B}(\mathcal{H})$. To show that the two measures $\rho\mu$ and $\nu$ coincide, we show that the corresponding characteristic functions are identical, that is

[TABLE]

For the measure $\nu$, the characteristic function is given by

[TABLE]

To compute the characteristic function for $\rho\mu$, we first note that by Lemma 12,

[TABLE]

Let $b_{k}=\frac{Q^{-1/2}(m_{2}-m_{1})}{(1-\alpha_{k})}$. The characteristic function for $\rho\mu$ is given by

[TABLE]

For each $k\in\mathbb{N}$, we have by Proposition 6, using the fact that $||\phi_{k}||=1$,

[TABLE]

For each $N\in\mathbb{N}$, for the function $s_{N}(x)=\exp\left[-\frac{1}{2}\sum_{k=1}^{N}\Phi_{k}(x)\right]$, we have by the independence of the $W_{\phi_{k}}$’s that

[TABLE]

By Proposition 7, there exists $p>1$ is such that $I+(p-1)S>0$. Then for $s_{N}(x)=\exp\left[-\frac{1}{2}\sum_{k=1}^{N}\Phi_{k}(x)\right]$, the sequence $\{s_{N}^{q}\}_{N\in\mathbb{N}}$ is uniformly integrable on $(\mathscr{B}(\mathcal{H}),\mu)$ for all $1\leq q<p$. Thus the sequence $\{\exp(iq\langle h,x\rangle)s_{N}^{q}(x)\}_{N\in\mathbb{N}}$ is also uniformly integrable for $1\leq q<p$. For $q=1$, Vitali’s Convergence Theorem gives

[TABLE]

For the first exponent, we have for any $h\in\mathcal{H}$,

[TABLE]

For the second exponent, since $\{\phi_{k}\}_{k\in\mathbb{N}}$ is an orthonormal basis for $\mathcal{H}$, we have

[TABLE]

For the third exponent,

[TABLE]

Thus, taking the limit as $N\rightarrow\infty$, we obtain

[TABLE]

Combining this with Eq. (94), we obtain the desired equality, namely

[TABLE]

∎

Lemma 13

Assume that $S$ is trace class. Then $\sum_{k=1}^{\infty}\frac{\alpha_{k}}{1-\alpha_{k}}W^{2}_{\phi_{k}}\in\mathcal{L}^{1}(\mathcal{H},\mu)$ and the following limit holds in the $\mathcal{L}^{1}(\mathcal{H},\mu)$ sense,

[TABLE]

Proof

We first note that, since $S$ is trace class, $S(I-S)^{-1}$ is also trace class and

[TABLE]

as $N\rightarrow\infty$. Furthermore,

[TABLE]

showing that $\sum_{k=1}^{\infty}\frac{\alpha_{k}}{1-\alpha_{k}}W^{2}_{\phi_{k}}\in\mathcal{L}^{1}(\mathcal{H},\mu)$. By Hölder’s Inequality, we have

[TABLE]

since $||W_{P_{N}\phi_{k}}||_{\mathcal{L}^{2}(\mathcal{H},\mu)}=||P_{N}\phi_{k}||\leq||\phi_{k}||=||W_{\phi_{k}}||_{\mathcal{L}^{2}(\mathcal{H},\mu)}=1$. It follows that

[TABLE]

as $N\rightarrow\infty$. ∎

Lemma 14

Let $b\in\mathcal{H}$ be arbitrary. Then $\sum_{k=1}^{\infty}\frac{1}{1-\alpha_{k}}W_{\phi_{k}}\langle b,\phi_{k}\rangle\in\mathcal{L}^{2}(\mathcal{H},\mu)$ and the following limit holds in the $\mathcal{L}^{2}(\mathcal{H},\mu)$ sense

[TABLE]

Proof

Since the sequence $\{W_{\phi_{k}}\}_{k\in\mathbb{N}}$ is orthonormal in $\mathcal{L}^{2}(\mathcal{H},\mu)$, we have

[TABLE]

Thus $\sum_{k=1}^{\infty}\frac{1}{1-\alpha_{k}}W_{\phi_{k}}\langle b,\phi_{k}\rangle\in\mathcal{L}^{2}(\mathcal{H},\mu)$. Furthermore,

[TABLE]

This gives the desired convergence. ∎

Proof

(of Corollary 2) When $S$ is trace class, the Fredholm determinant $\det(I-S)$ is well-defined and for $I-S$ strictly positive, we have

[TABLE]

From the spectral decomposition $S(I-S)^{-1}=\sum_{k=1}^{\infty}\frac{\alpha_{k}}{1-\alpha_{k}}\phi_{k}\otimes\phi_{k}$, we have $\forall N\in\mathbb{N}$,

[TABLE]

By Lemma 13, taking limit as $N\rightarrow\infty$ gives, where the limit is in $\mathcal{L}^{1}(\mathcal{H},\mu)$,

[TABLE]

Similarly,

[TABLE]

By Lemma 14, taking limit as $N\rightarrow\infty$, we have

[TABLE]

Combining these, we obtain

[TABLE]

∎

6.1 Exact Kullback-Leibler divergences

We now derive the explicit expression for the exact Kullback-Leibler divergence between two equivalent Gaussian measures on $\mathcal{H}$. In the following, let $\mu=\mathcal{N}(m_{1},Q)$ and $W:\mathcal{H}\rightarrow\mathcal{L}^{2}(\mathcal{H},\mu)$ be the white noise mapping induced by $\mu$. Let $\nu=\mathcal{N}(m_{2},R)$, with $m_{2}-m_{1}\in{\rm Im}(Q^{1/2})$ and $R=Q^{1/2}(I-S)Q^{1/2}$ for some $S\in{\rm Sym}(\mathcal{H})\cap{\rm HS}(\mathcal{H})$. Let $\{\alpha_{k}\}_{k\in\mathbb{N}}$ be the eigenvalues of $S$ with corresponding orthonormal eigenvectors $\{\phi_{k}\}_{k\in\mathbb{N}}$.

Theorem 6.5

*Let $\mu=\mathcal{N}(m_{1},Q)$ and $\nu=\mathcal{N}(m_{2},R)$, with $m_{2}-m_{1}\in{\rm Im}(Q^{1/2})$ and $R=Q^{1/2}(I-S)Q^{1/2}$, where $S\in{\rm Sym}(\mathcal{H})\cap{\rm HS}(\mathcal{H})$. Then *

[TABLE]

If, furthermore, $S$ is trace class, then

[TABLE]

For $m_{1}=m_{2}=0$, we obtain the Kullback-Leibler divergence given in Michalek:1999 , which also derived the Rényi divergences between two zero-mean Gaussian measures with different covariance operators.

Lemma 15

For any $z,z_{1},z_{2}\in\mathcal{H}$,

[TABLE]

In particular, for the orthonormal eigenvectors $\{\phi_{k}\}_{k\in\mathbb{N}}$ of $S$,

[TABLE]

Proof

For $z\in Q^{1/2}(\mathcal{H})$, which is dense in $\mathcal{H}$, we have

[TABLE]

By a limiting argument, we then have $\int_{\mathcal{H}}W_{z}(x)d\nu(x)=\langle Q^{-1/2}(m_{2}-m_{1}),z\rangle$ $\forall z\in\mathcal{H}$.

For any pair $(z_{1},z_{2})\in Q^{1/2}(\mathcal{H})$, we have

[TABLE]

[TABLE]

Since $Q^{1/2}(\mathcal{H})$ is dense in $\mathcal{H}$, by a limiting argument, we have $\forall z_{1},z_{2}\in\mathcal{H}$,

[TABLE]

For the orthonormal basis $\{\phi_{k}\}_{k\in\mathbb{N}}$, we have $\langle(I-S)\phi_{j},\phi_{k}\rangle=(1-\alpha_{k})\delta_{jk}$, so that

[TABLE]

∎

Proposition 8

*Consider the functions *

[TABLE]

Then $g\in\mathcal{L}^{1}(\mathcal{H},\nu)$,$g\in\mathcal{L}^{2}(\mathcal{H},\nu)$, and

[TABLE]

Proof

Using the expression for $\langle W_{\phi_{j}},W_{\phi_{k}}\rangle_{\mathcal{L}^{2}(\mathcal{H},\nu)}$ from Lemma 15, we have

[TABLE]

Furthermore, the expression for $||g_{N}||^{2}_{L^{2}(\mathcal{H},\nu)}$ shows that

[TABLE]

By the Hölder Inequality, we obtain $||g||_{\mathcal{L}^{1}(\mathcal{H},\nu)}\leq||g||_{\mathcal{L}^{2}(\mathcal{H},\nu)}<\infty$ and $||g_{N}-g||_{\mathcal{L}^{1}(\mathcal{H},\nu)}\leq||g_{N}-g||_{\mathcal{L}^{2}(\mathcal{H},\nu)}\rightarrow 0\;\;\text{as}\;\;N\rightarrow\infty$. ∎

Lemma 16

For any pair $a_{1},a_{2}\in\mathcal{H}$,

[TABLE]

In particular, for $a_{1}=a_{2}=a$,

[TABLE]

Proof

We have, by symmetry, for any $a\in\mathcal{H}$, $\int_{\mathcal{H}}\langle x-m_{2},a\rangle\mathcal{N}(m_{2},R)=\int_{\mathcal{H}}\langle x-m_{2},a\rangle^{3}\mathcal{N}(m_{2},R)=0$. Also, by Lemma 20, for any $a,b\in\mathcal{H}$,

[TABLE]

Thus for any pair $a_{1},a_{2}\in\mathcal{H}$, by Lemma 19,

[TABLE]

This completes the proof. ∎

Lemma 17

For any pair $a,b\in Q^{1/2}(\mathcal{H})$,

[TABLE]

Proof

By assumption, there exist $c,d\in\mathcal{H}$ such that $a=Q^{1/2}c$, $b=Q^{1/2}d$. Thus

[TABLE]

∎

Lemma 18

For any $a,b\in\mathcal{H}$,

[TABLE]

In particular, for $a=b$,

[TABLE]

For two orthonormal eigenvectors $\phi_{k},\phi_{j}$ of $S$,

[TABLE]

Proof

For $a,b\in Q^{1/2}(\mathcal{H})$, by Lemmas 16 and 17, we have

[TABLE]

The general case $a,b\in\mathcal{H}$ then follows by a limiting argument. ∎

Proposition 9

The following functions are orthonormal in $\mathcal{L}^{2}(\mathcal{H},\nu)$

[TABLE]

Proof

We have by Lemma 15 that

[TABLE]

Thus the constant function $1$ is orthogonal to $W_{\phi_{k}}^{2}-[1-\alpha_{k}+\langle Q^{-1/2}(m_{2}-m_{1}),\phi_{k}\rangle^{2}]$. By Lemma 18, for $k\neq j\in\mathbb{N}$,

[TABLE]

thus the sequence $\{W^{2}_{\phi_{k}}(x)-\int_{\mathcal{H}}W_{\phi_{k}}^{2}(x)\mathcal{N}(m_{2},R)\}_{k\in\mathbb{N}}$ is orthogonal. By Lemma 18,

[TABLE]

This gives the normalization constant for each term in the sequence. ∎

Proposition 10

Consider the functions

[TABLE]

Then $f\in\mathcal{L}^{1}(\mathcal{H},\nu)$, $f\in\mathcal{L}^{2}(\mathcal{H},\nu)$, and

[TABLE]

Proof

Let $a_{k}=[(1-\alpha_{k})^{2}+2(1-\alpha_{k})\langle Q^{-1/2}(m_{2}-m_{1}),\phi_{k}\rangle^{2}]$, $b_{k}=[1-\alpha_{k}+\langle Q^{-1/2}(m_{2}-m_{1}),\phi_{k}\rangle^{2}]$, then

[TABLE]

Consider the series of constants

[TABLE]

Consider the functions

[TABLE]

By Proposition 9 and the definition of $a_{k}$ above, we have

[TABLE]

Thus $h\in\mathcal{L}^{2}(\mathcal{H},\nu)$. Furthermore,

[TABLE]

as $N\rightarrow\infty$. Thus it follows that $f\in\mathcal{L}^{2}(\mathcal{H},\nu)$ and $\lim_{N\rightarrow\infty}||f_{N}-f||_{\mathcal{L}^{2}(\mathcal{H},\nu)}=0$. Since $\nu$ is a probability measure on $\mathcal{H}$, it also follows that $f\in\mathcal{L}^{1}(\mathcal{H},\nu)$ and that $\lim_{N\rightarrow\infty}||f_{N}-f||_{\mathcal{L}^{1}(\mathcal{H},\nu)}=0$. This completes the proof. ∎

Proof

(of Theorem 6.5) By Theorem 6.2,

[TABLE]

For each $k\in\mathbb{N}$, by Lemma 15, we obtain

[TABLE]

For each $N\in\mathbb{N}$, consider the function $r_{N}=f_{N}-2g_{N}$, $r=f-2g$, where

[TABLE]

By Propositions 10 and 8, we have $f\in\mathcal{L}^{1}(\mathcal{H},\nu)$, $g\in\mathcal{L}^{1}(\mathcal{H},\nu)$, and

[TABLE]

It follows that $r\in\mathcal{L}^{1}(\mathcal{H},\nu)$ and that $\lim_{N\rightarrow\infty}||r_{N}-r||_{\mathcal{L}^{1}(\mathcal{H},\nu)}=0$. Therefore

[TABLE]

Combining the last expression with the expression for $\log\left\{\frac{d\nu}{d\mu}(x)\right\}$, we obtain

[TABLE]

∎

Proof (**

of Theorem 3.3**)

Consider the formula

[TABLE]

where $S$ is given by $S=C_{0}^{1/2}A^{*}(AC_{0}A^{*}+\Gamma)^{-1}AC_{0}^{1/2}\in{\rm Tr}(\mathcal{H})$. By Theorem 6.5,

[TABLE]

For the first term, since $m=m_{0}+C_{0}A^{*}(AC_{0}A^{*}+\Gamma)^{-1}(y-Am_{0})$,

[TABLE]

For the second and third terms,

[TABLE]

From the expression $C=C_{0}-C_{0}A^{*}(AC_{0}A^{*}+\Gamma)^{-1}AC_{0}$, we obtain

[TABLE]

Thus we have

[TABLE]

For the term $\log\det(I-S)$, we have

[TABLE]

from which it follows that $\log\det(I-S)=\log\det(\Gamma)-\log\det(AC_{0}A^{*}+\Gamma)$. Combining $\log\det(I-S)$ and ${\rm tr}(S)$ with the first term gives the desired result. ∎

6.2 Exact Rényi divergences

In this section, we derive the exact formula for the Rényi divergences $D_{{\rm R},r}(\nu||\mu)$ between two equivalent Gaussian measures $\nu$ and $\mu$ on $\mathcal{H}$. We recall that the Rényi divergence between $\nu$ and $\mu$ is defined by

[TABLE]

Theorem 6.6

Let $\mu=\mathcal{N}(m_{1},Q)$, $\nu=\mathcal{N}(m_{2},R)$, with $m_{2}-m_{1}\in{\rm Im}(Q^{1/2})$ and $R=Q^{1/2}(I-S)Q^{1/2}$, $S\in{\rm Sym}(\mathcal{H})\cap{\rm HS}(\mathcal{H})$. The Rényi divergence of order $r$, $0<r<1$, between $\nu$ and $\mu$ is given by

[TABLE]

Furthermore,

[TABLE]

Proof (**

of Theorem 6.6**)

By Proposition 7, there exists $p>1$ such that $I+(p-1)S>0$. Proposition 7 then implies that $\frac{d\nu}{d\mu}\in\mathcal{L}^{q}(\mathcal{H},\mu)$ for all $q$ satisfying $0<q<p$. By definition of the Rényi divergence, we then have for $0<r<1$,

[TABLE]

By Proposition 7, we have for $0<r<1$,

[TABLE]

Thus it follows that

[TABLE]

Let $c=Q^{-1/2}(m_{2}-m_{1})$, then we have

[TABLE]

Combining this with the previous expression, we obtain

[TABLE]

This completes the proof of the first part of the theorem.

We now compute $\lim_{r\rightarrow 1^{-}}D_{{\rm R},r}(\nu||\mu)$. Let $\{\alpha_{k}\}_{k\in\mathbb{N}}$ be the eigenvalues of $S$, then

[TABLE]

By Lemma 23, we have $\frac{1}{1-r}[\log(1-(1-r)\alpha_{k})-(1-r)\log(1-\alpha_{k})]\geq 0$ $\forall k\in\mathbb{N}$. Thus by Lebesgue’s Monotone Convergence Theorem,

[TABLE]

Combining this limit with the expression for $D_{{\rm R},r}(\nu||\mu)$ above, we obtain

[TABLE]

Also by Lemma 23 and Lebesgue’s Monotone Convergence Theorem,

[TABLE]

From the proof of Theorem 5.1, we have for any $m\in{\rm Im}(Q^{1/2})$,

[TABLE]

Combining the previous two limits with the expression for $D_{{\rm R},r}(\nu||\mu)$ above, we obtain

[TABLE]

∎

6.3 Bhattacharyya and Hellinger distances

We now derive the explicit formulas for the Bhattacharyya and Hellinger distances between two equivalent Gaussian measures $\nu$ and $\mu$ on $\mathcal{H}$. Recall that the Bhattacharyya distance is defined by

[TABLE]

The Hellinger distance $D_{{\rm H}}(\nu||\mu)$ between $\nu$ and $\mu$ is defined by

[TABLE]

Corollary 3

Let $\mu=\mathcal{N}(m_{1},Q)$ and $\nu=\mathcal{N}(m_{2},R)$ and $S\in{\rm Sym}(\mathcal{H})\cap{\rm HS}(\mathcal{H})$ be such that $m_{2}-m_{1}\in{\rm Im}(Q^{1/2})$ and $R=Q^{1/2}(I-S)Q^{1/2}$. The Bhattacharyya distance $D_{{\rm B}}(\nu||\mu)$ between $\nu$ and $\mu$ is then given by

[TABLE]

The Hellinger distance $D_{{\rm H}}(\nu||\mu)$ between $\nu$ and $\mu$ is given by

[TABLE]

Proof (**

of Corollary 3**)

For the Bhattacharyya distance, we use the fact that $D_{{\rm B}}(\nu||\mu)=\frac{1}{4}D_{{\rm R},1/2}(\nu||\mu)$ and Theorem 6.6 to obtain

[TABLE]

The expression for $D_{{\rm H}}(\nu||\mu)$ then follows from $D_{{\rm H}}^{2}(\nu||\mu)=2[1-\exp(-D_{{\rm B}}(\nu||\mu))]$. ∎

Proof (**

of Theorems 3.1 and 3.2 and Corollary 1**)

Theorems 3.1 follows from Theorem 4.1 and Theorem 6.5. Theorem 3.2 follows from Theorem 5.1 and Theorem 6.6. Corollary 1 follows from Theorem 5.1 and Corollary 3. ∎

7 Miscellaneous technical results

Let $\mathcal{N}(m,Q)$ denote a Gaussian measure on $\mathcal{H}$ with mean $m$ and covariance operator $Q$. Let $\{\lambda_{k}\}_{k=1}^{\infty}$ denote the set of eigenvalues of $Q$, with corresponding orthonormal eigenvectors $\{e_{k}\}_{k=1}^{\infty}$.

Lemma 19

For any pair $a,b\in\mathcal{H}$,

[TABLE]

In particular, for $a=b$, $\int_{\mathcal{H}}\langle x-m,a\rangle^{4}\mathcal{N}(m,Q)(dx)=3\langle a,Qa\rangle^{2}$.

Proof

It suffices to prove for $m=0$. We apply the following (Handbook:1972 , Formula 7.4.4)

[TABLE]

Thus for any $\lambda>0$,

[TABLE]

Write $x=\sum_{k=1}^{\infty}x_{k}e_{k}$, $a=\sum_{k=1}^{\infty}a_{k}e_{k}$. By symmetry, we have

[TABLE]

∎

Lemma 20

For any pair $a,b\in\mathcal{H}$,

[TABLE]

In particular, for $a=b$, $\int_{\mathcal{H}}\langle x-m,a\rangle^{3}\mathcal{N}(m,Q)(dx)=0$.

Proof

It suffices to prove for $m=0$. Write $x=\sum_{k=1}^{\infty}x_{k}e_{k}$, $a=\sum_{k=1}^{\infty}a_{k}e_{k}$, then

[TABLE]

by symmetry, since each term in the integral contains either $x_{j}$ or $x_{j}^{3}$ $\forall k\in\mathbb{N}$. ∎

Lemma 21

In all inequalities below, equality happens if and only if $x=0$.

[TABLE]

Lemma 22

Let $p>1$ be fixed. Then

[TABLE]

Lemma 23

Let $\alpha<1$ be fixed. Then

[TABLE]