Divergence functions in Information Geometry

TL;DR

This paper explores the properties of a canonical divergence function within Information Geometry, comparing it to other divergences and discussing open problems related to its symmetry features.

Contribution

It introduces and analyzes a canonical divergence in Information Geometry, highlighting its relation to existing divergences and outlining open problems about its symmetry.

Findings

Connections between the canonical divergence and other divergence functions

Discussion of symmetry properties and open problems

Insights into the structure of dual connections in Information Geometry

Abstract

A recently introduced canonical divergence $\mathcal{D}$ for a dual structure $(\mathrm{g},\nabla,\nabla^*)$ is discussed in connection to other divergence functions. Finally, open problems concerning symmetry properties are outlined.

Tables

Table 1. Table 1: The column on the left describes the category of statistical manifolds on which the canonical divergence 𝒟 𝒟 \displaystyle\mathcal{D} reduces to the divergence D D \displaystyle\mathrm{D} . The column on the right shows the properties of the curvature tensor characterizing the corresponding manifolds and supplying the equivalence 𝒟 ≡ D 𝒟 D \displaystyle\mathcal{D}\equiv\mathrm{D} through the Eq. ( 20 ). X , Y ∈ 𝒯 ​ ( M ) 𝑋 𝑌 𝒯 M \displaystyle X,Y\in\mathcal{T}(\mathrm{M}) .

Statistical manifold			Condition on $ℛ$
Self-dual			$ℛ(X,X)=0$
Dually flat			$ℛ\equiv 0$
Symmetric			$\nabla ℛ\equiv 0,R(Y,X,X,X)=0$

Table 2. Table 2: The symmetry property of 𝒟 𝒟 \displaystyle\mathcal{D} fails for general statistical manifolds. However, it holds true for dually flat and statistical symmetric manifolds Felice18 .

Statistical manifold			Relation of $𝒟(q,p)$ and ${𝒟}^{\ast }(p,q)$
Dually flat			$𝒟(q,p)={𝒟}^{\ast }(p,q)$
Symmetric			$𝒟(q,p)={𝒟}^{\ast }(p,q)$
General			$𝒟(q,p)=f\left({𝒟}^{\ast }(p,q)\right)$

Equations

$$\mathcal{D}(p,q)\ge 0,\mathcal{D}(p,q)=0\text{mbox}iffp=q.$$

$$\mathrm{K}(p,q)=\sum _{i=1}^{n+1}{p}_i\log \left(\frac{p_i}{q_i}\right)$$

$${{\partial }_i\mathcal{D}({\mathbf{\xi }}_p,{\mathbf{\xi }}_q)|}_{p=q}=0,{{\partial }_i^{\prime }\mathcal{D}({\mathbf{\xi }}_p,{\mathbf{\xi }}_q)|}_{p=q}=0$$

$${{\partial }_j{\partial }_i\mathcal{D}({\mathbf{\xi }}_p,{\mathbf{\xi }}_q)|}_{p=q}=-{{\partial }_j^{\prime }{\partial }_i\mathcal{D}({\mathbf{\xi }}_p,{\mathbf{\xi }}_q)|}_{p=q}={{\partial }_j^{\prime }{\partial }_i^{\prime }\mathcal{D}({\mathbf{\xi }}_p,{\mathbf{\xi }}_q)|}_{p=q},$$

$${\mathrm{g}}_{ij}:={{\partial }_i{\partial }_j\mathcal{D}({\mathbf{\xi }}_p,{\mathbf{\xi }}_q)|}_{p=q}>0$$

$${\Gamma }_{ijk}(p)=-{{\partial }_i{\partial }_j{\partial }_k^{\prime }\mathcal{D}({\mathbf{\xi }}_p,{\mathbf{\xi }}_q)|}_{p=q},{\Gamma }_{ijk}^{\ast }(p)=-{{\partial }_i^{\prime }{\partial }_j^{\prime }{\partial }_k\mathcal{D}({\mathbf{\xi }}_p,{\mathbf{\xi }}_q)|}_{p=q}.$$

$$X\mathrm{g}\left(Y,Z\right)=\mathrm{g}\left({\nabla }_{X}Y,Z\right)+\mathrm{g}\left(Y,{\nabla }_X^{\ast }Z\right),\forall X,Y,Z\in \mathcal{T}(\mathrm{M}),$$

$${\mathrm{X}}_q(p):=\mathrm{X}(q,p):={\exp }_q^{-1}(p)=\dot{\tilde{\sigma }}(0),$$

$$\mathrm{D}(p,q):={\int }_0^1⟨{\mathrm{X}}_t(p),\dot{\tilde{\sigma }}(t){⟩}_{\tilde{\sigma }(t)}dt,{\mathrm{X}}_t(p):=\mathrm{X}(\tilde{\sigma }(t),p):={\exp }_{\tilde{\sigma }(t)}^{-1}(p),$$

$$\mathrm{D}(p,q)={\int }_0^{1}t\Vert \dot{\sigma }(t){\Vert }^{2}dt,$$

$${\Pi }_q(p):={\mathrm{P}}_{{\sigma }^{\ast }}{\mathrm{X}}_p(q)\in {\mathrm{T}}_q\mathrm{M}.$$

$${\Pi }_q^{\ast }(p):={\mathrm{P}}_{\sigma }^{\ast }{\mathrm{X}}_p^{\ast }(q),{\mathrm{X}}_p^{\ast }(q):={\stackrel{\ast }{\exp }}_p^{-1}(q)={\dot{\sigma }}^{\ast }(0),$$

$${\Pi }_q(p)+{\Pi }_q^{\ast }(p)={\mathrm{grad}}_q{r}_p,r(p,q):=⟨{\exp }_p^{-1}(q),{\stackrel{\ast }{\exp }}_p^{-1}(q){⟩}_p.$$

$${\Pi }_t(p)={\mathrm{P}}_{{\sigma }_t^{\ast }}{\mathrm{X}}_p(t),$$

$${\Pi }_t^{\ast }(p)={\mathrm{P}}_{{\sigma }_t}^{\ast }{\mathrm{X}}_p^{\ast }(t),$$

$${\int }_0^1⟨{\Pi }_t(p),\dot{\gamma }(t){⟩}_{\gamma (t)}dt+{\int }_0^1⟨{\Pi }_t^{\ast }(p),\dot{\gamma }(t){⟩}_{\gamma (t)}dt=r_p(q)$$

$$\mathcal{D}(p,q):={\mathcal{D}}_p(q):={\int }_0^1⟨{\Pi }_t(p),\dot{\sigma }(t){⟩}_{\sigma (t)}\mathrm{d}t,$$

$${\mathcal{D}}^{\ast }(p,q):={\mathcal{D}}_p^{\ast }(q):={\int }_0^1⟨{\Pi }_t^{\ast }(p),{\dot{\sigma }}^{\ast }(t){⟩}_{{\sigma }^{\ast }(t)}\mathrm{d}t,$$

$${\mathrm{P}}_{{\Sigma }_t}{\mathrm{X}}_p(t)={\mathrm{X}}_p(t)+{\mathcal{R}}_{{\Sigma }_t}\left({\mathrm{X}}_p^{\ast }(t),{\mathrm{X}}_p(t)\right),$$

$${\mathcal{R}}_{{\Sigma }_t}\left({\mathrm{X}}_p^{\ast }(t),{\mathrm{X}}_p(t)\right):={\int }_{B_t}\frac{\mathrm{P}\left[\mathcal{R}\left({\mathrm{X}}^{\ast }(t),\mathrm{X}(t)\right)\mathrm{X}(t)\right]}{\Vert {\mathrm{X}}_p^{\ast }(t)\wedge {\mathrm{X}}_p(t)\Vert }\mathrm{d}A$$

$${\Pi }_t(p)$$

$$\mathcal{D}(p,q)=\mathrm{D}(p,q)+{\int }_0^1⟨{\mathrm{P}}_{{\sigma }_t}\left[{\mathcal{R}}_{{\Sigma }_t}\left({\mathrm{X}}_p^{\ast }(t),{\mathrm{X}}_p(t)\right)\right],\dot{\sigma }(t){⟩}_{\sigma (t)}\mathrm{d}t,$$

$${\Pi }_t(p)={\mathrm{grad}}_{\gamma (t)}{\mathcal{D}}_p(\gamma (t))+V_t,⟨V_t,{\dot{\sigma }}_t(1){⟩}_{\gamma (t)}=0$$

$${\Pi }_t^{\ast }(p)={\mathrm{grad}}_{\gamma (t)}{\mathcal{D}}_p^{\ast }(\gamma (t))+V_t^{\ast },⟨V_t^{\ast },{\dot{\sigma }}_t^{\ast }(1){⟩}_{\gamma (t)}=0,$$

$$\mathcal{H}(\kappa )=\{q\in \mathrm{M}|{\mathcal{D}}_p(q)=\kappa \},{\mathcal{H}}^{\ast }(\kappa )=\{q\in \mathrm{M}|{\mathcal{D}}_p^{\ast }(q)=\kappa \}.$$

$$L_q:=\{p\in \mathrm{M}|\mathcal{D}(p,q)=\underset{q^{\prime }\in \mathcal{H}}{\min }\mathcal{D}(p,q^{\prime })\}.$$

$${\partial }_U^{\prime }\mathcal{D}(p,q)=0,\forall U\in {\mathrm{T}}_q\mathcal{H},p\in L_q,$$

$$\mathrm{g}\left(II({\rm Y},\Xi ),U\right):=-{{\partial }_{\Xi }{\partial }_{{\rm Y}}{\partial }_U^{\prime }\mathcal{D}(p,q)|}_{p=q}=0,\forall \Xi ,{\rm Y}\in {\mathrm{T}}_q{L}_q,U\in {\mathrm{T}}_q\mathcal{H},$$

$${\nabla }_{\Xi }{\rm Y}={\nabla }_{\Xi }^{L_q}{\rm Y}+II(\Xi ,{\rm Y})$$

$$\mathcal{D}(q,p)=f\left({\mathcal{D}}^{\ast }(p,q)\right).$$

Full text

I Introduction

The geometrical structure induced by a divergence function (or contrast function) on a smooth manifold $\displaystyle\mathrm{M}$ provides a unified approach to measurement of notions as information, energy, entropy, playing an important role in mathematical sciences to research random phenomena Eguchi92 . In the mathematical formulation, a divergence function $\displaystyle\mathcal{D}(p,q)$ on a smooth manifold $\displaystyle\mathrm{M}$ is defined by the first requirement for a distance:

[TABLE]

An important example of a divergence function is given by the Kullback-Leibler divergence $\displaystyle\mathrm{K}(p,q)$ in the context that $\displaystyle p$ and $\displaystyle q$ are the vectors of probabilities of disjoint events Eguchi85 , namely

[TABLE]

is a function on the $\displaystyle n$-simplex $\displaystyle\mathcal{S}:=\{p=(p_{1},\ldots,p_{n+1})\,|\,p_{i}\geq 0\,,\sum_{i}p_{i}=1\}$. Given a smooth $\displaystyle n$-dimensional manifold $\displaystyle\mathrm{M}$, we assume that $\displaystyle\mathcal{D}:\mathrm{M}\times\mathrm{M}\rightarrow\mathbb{R}^{+}$ is a $\displaystyle C^{\infty}$-differentiable function. Working with the local coordinates $\displaystyle\{\boldsymbol{\xi}_{p}:=(\xi_{p}^{1},\ldots,\xi^{n}_{p})\}$ and $\displaystyle\{\boldsymbol{\xi}_{q}:=(\xi_{q}^{1},\ldots,\xi^{n}_{q})\}$ at $\displaystyle p$ and $\displaystyle q$, respectively, it follows from Eq. (1) that

[TABLE]

where $\displaystyle\partial_{i}=\frac{\partial}{\partial\xi^{i}_{p}}$ and $\displaystyle\partial_{i}^{\prime}=\frac{\partial}{\partial\xi^{i}_{q}}$. Moreover, under the assumption that

[TABLE]

we can see that the manifold $\displaystyle\mathrm{M}$ is endowed, through the divergence function $\displaystyle\mathcal{D}$, with the Riemannian metric tensor given by $\displaystyle{\rm g}={\rm g}_{ij}\,\mathrm{d}\xi^{i}\otimes\mathrm{d}\xi^{j}$, where the Einstein notation is adopted. The symmetry of $\displaystyle{\rm g}$ immediately follows from the requirement that $\displaystyle\mathcal{D}$ is a $\displaystyle C^{\infty}$ function on $\displaystyle\mathrm{M}\times\mathrm{M}$.

From Eq. (2) we can see that, in general, a divergence function $\displaystyle\mathcal{D}$ is not symmetric. The asymmetry of $\displaystyle\mathcal{D}$ leads to two different affine connections, $\displaystyle\nabla$ and $\displaystyle\nabla^{*}$, on $\displaystyle\mathrm{M}$ such that $\displaystyle 1/2(\nabla+\nabla^{*})$ is the Levi-Civita connection with respect to the metric tensor $\displaystyle{\rm g}={\rm g}_{ij}\mathrm{d}\xi^{i}\otimes\mathrm{d}\xi^{j}$ defined by Eq. (5). More precisely, working with the local coordinates $\displaystyle\{\boldsymbol{\xi}_{p}\}$ and $\displaystyle\{\boldsymbol{\xi}_{q}\}$, we can define the symbols $\displaystyle\Gamma_{ijk}$ and $\displaystyle\Gamma_{ijk}^{*}$ of the connections $\displaystyle\nabla$ and $\displaystyle\nabla^{*}$, i.e. $\displaystyle\Gamma_{ijk}={\rm g}\left(\nabla_{\partial_{i}}\partial_{j},\partial_{k}\right)$ and $\displaystyle{\Gamma}^{*}_{ijk}={\rm g}\left(\nabla^{*}_{\partial_{i}}\partial_{j},\partial_{k}\right)$, by means of the following relations

[TABLE]

To sum up, a divergence function $\displaystyle\mathcal{D}$ on a smooth manifold $\displaystyle\mathrm{M}$ induces a metric tensor on $\displaystyle\mathrm{M}$ by Eq. (5). In addition, the divergence $\displaystyle\mathcal{D}$ yields two linear torsion-free connections, $\displaystyle\nabla$ and $\displaystyle\nabla^{*}$, on $\displaystyle{\rm T}\mathrm{M}$ which are dual with respect to the metric tensor $\displaystyle{\rm g}$ Eguchi85 :

[TABLE]

where $\displaystyle\mathcal{T}(\mathrm{M})$ denotes the space of vector fields on $\displaystyle\mathrm{M}$. Finally, we refer to the quadruple $\displaystyle(\mathrm{M},{\rm g},\nabla,\nabla^{*})$ as a statistical manifold Ay17 .

I.1 The inverse problem within Information Geometry

The inverse problem is to find a divergence $\displaystyle\mathcal{D}$ which generates a given geometrical structure $\displaystyle(\mathrm{M},{\rm g},\nabla,\nabla^{*})$. For any such statistical manifold there exists a divergence $\displaystyle\mathcal{D}$ such that Eq. (5) and Eq. (6) hold true Matumoto93 . However, this divergence is not unique and there are infinitely many divergences generating the same geometrical structure $\displaystyle({\rm g},\nabla,\nabla^{*})$. When this structure is dually flat, namely the curvature tensors of $\displaystyle\nabla$ and $\displaystyle\nabla^{*}$ are null ($\displaystyle\mathrm{R}(\nabla)\equiv 0$ and $\displaystyle\mathrm{R}^{*}(\nabla^{*})\equiv 0$), Amari and Nagaoka introduced a canonical divergence which is a Bregman divergence Amari00 . The canonical divergence has nice properties such as the generalized Pythagorean theorem and the geodesic projection theorem and it turns out to be of uppermost importance to define a canonical divergence in the general case. A first attempt to answer this fundamental issue is provided by Ay and Amari in Ay15 where a canonical divergence for a general statistical manifold $\displaystyle(\mathrm{M},{\rm g},\nabla,\nabla^{*})$ is given by using the geodesic integration of the inverse exponential map. This one is understood as a difference vector that translates $\displaystyle q$ to $\displaystyle p$ for all $\displaystyle q,p\in\mathrm{M}$ sufficiently close to each other.

To be more precise, the inverse exponential map supplies a generalization to $\displaystyle\mathrm{M}$ of the concept of difference vector in $\displaystyle\mathbb{R}^{n}$. In detail, let $\displaystyle p,q\in\mathbb{R}^{n}$, the difference between $\displaystyle p$ and $\displaystyle q$ is the vector $\displaystyle p-q$ pointing to $\displaystyle p$ (see side ($\displaystyle\mathbf{A}$) of Fig. 1). Then, the difference between $\displaystyle p$ and $\displaystyle q$ in $\displaystyle\mathrm{M}$ is provided by the inverse exponential map. In particular, given $\displaystyle p,q$ suitably close in $\displaystyle\mathrm{M}$, the difference vector from $\displaystyle q$ to $\displaystyle p$ is defined as (see ($\displaystyle\mathbf{B}$) of Fig. 1)

[TABLE]

where $\displaystyle\widetilde{\sigma}$ is the $\displaystyle\nabla$-geodesic from $\displaystyle q$ to $\displaystyle p$. Therefore, the divergence $\displaystyle\mathrm{D}$ introduced in Ay15 is defined as the path integral

[TABLE]

where $\displaystyle\langle\cdot,\cdot\rangle_{\widetilde{\sigma}(t)}$ denotes the inner product induced by $\displaystyle{\rm g}$ on $\displaystyle\widetilde{\sigma}(t)$. After elementary computation Eq. (9) reduces to,

[TABLE]

where $\displaystyle\sigma(t)$ is the $\displaystyle\nabla$-geodesic from $\displaystyle p$ to $\displaystyle q$ Ay15 . The divergence $\displaystyle\mathrm{D}(p,q)$ has nice properties such as the positivity and it reduces to the canonical divergence proposed by Amari and Nagaoka when the manifold $\displaystyle\mathrm{M}$ is dually flat. However, if we consider definition (9) for a general path $\displaystyle\gamma$, then $\displaystyle\mathrm{D}_{\gamma}(p,q)$ will be depending on $\displaystyle\gamma$. On the contrary, if the vector field $\displaystyle\mathrm{X}_{t}(p)$ is integrable, then $\displaystyle\mathrm{D}_{\gamma}(p,q)=:\mathrm{D}(p,q)$ turns out to be independent of the path from $\displaystyle q$ to $\displaystyle p$.

II A new canonical divergence

In this article, we discuss about a novel divergence function recently introduced in Felice18 . This turns out to be a generalization of the divergence introduced by Ay and Amari. The definition of the new divergence (see below Eqs. (17), (18)) relies on an extended analysis of the intrinsic structure of the dual geometry of a general statistical manifold $\displaystyle(\mathrm{M},{\rm g},\nabla,\nabla^{*})$. In particular, we introduced a vector at $\displaystyle q$ by modifying the definition (8) of the difference vector $\displaystyle\mathrm{X}(q,p)$. Consider $\displaystyle p,q\in\mathrm{M}$ such that there exist both, a unique $\displaystyle\nabla$-geodesic $\displaystyle\sigma$ and a unique $\displaystyle\nabla^{*}$-geodesic $\displaystyle\sigma^{*}$, connecting $\displaystyle p$ to $\displaystyle q$. Moreover, let $\displaystyle\mathrm{X}_{p}(q):=\exp_{p}^{-1}(q)=\dot{\sigma}(0)\in{\rm T}_{p}\mathrm{M}$, we then $\displaystyle\nabla$-parallel translate it along $\displaystyle\sigma^{*}$ from $\displaystyle p$ to $\displaystyle q$ (see ($\displaystyle\mathbf{C}$) of Fig. 1), and obtain

[TABLE]

(Note that $\displaystyle\Pi_{q}(p)$ corresponds to minus a difference vector.) Analogously, we introduce the dual vector of $\displaystyle\Pi_{q}(p)$ as the $\displaystyle\nabla^{*}$-parallel transport of $\displaystyle\dot{\sigma}^{*}(0)$ along the $\displaystyle\nabla$-geodesic $\displaystyle\sigma$,

[TABLE]

where $\displaystyle\stackrel{{\scriptstyle*}}{{\exp}}_{p}$ denotes the exponential map of the $\displaystyle\nabla^{*}$-connection. A fundamental result obtained in Felice18 is that the sum of $\displaystyle\Pi_{q}(p)$ and $\displaystyle\Pi^{*}_{q}(p)$ is the gradient of a symmetric function $\displaystyle r_{p}(q)=r(p,q)$ that we call pseudo-norm:

[TABLE]

By letting $\displaystyle q\in\mathrm{M}$ be varying, we then obtain two vector fields whenever $\displaystyle p$ and $\displaystyle q$ are connected by a unique $\displaystyle\nabla$-geodesic and a unique $\displaystyle\nabla^{*}$-geodesic. Then, we can introduce two vector fields on an arbitrary path $\displaystyle\gamma:[0,1]\rightarrow\mathrm{M}$ connecting $\displaystyle p$ and $\displaystyle q$ in the following way. Let us firstly assume that for each $\displaystyle t\in[0,1]$ there exist a unique $\displaystyle\nabla$-geodesic $\displaystyle\sigma_{t}$ and a unique $\displaystyle\nabla^{*}$-geodesic $\displaystyle\sigma^{*}_{t}$ connecting $\displaystyle p$ with $\displaystyle\gamma(t)$. Then, we define

[TABLE]

Therefore, from Eq. (13) we have that the sum

[TABLE]

is independent of the particular path from $\displaystyle p$ to $\displaystyle q$.

At this point, we define the novel canonical divergence $\displaystyle\mathcal{D}$, and its dual function $\displaystyle\mathcal{D}^{*}$, from $\displaystyle p$ to $\displaystyle q$ by the geodesic integration of $\displaystyle\Pi_{t}(p)$ and $\displaystyle\Pi^{*}_{t}(p)$, respectively. In particular, we have that

[TABLE]

where $\displaystyle\sigma$ and $\displaystyle\sigma^{*}$ are the $\displaystyle\nabla$-geodesic and the $\displaystyle\nabla^{*}$-geodesic from $\displaystyle p$ to $\displaystyle q$, respectively.

In this manuscript we review the relation of the canonical divergence $\displaystyle\mathcal{D}(p,q)$ to other divergence functions in Section III. Finally, we outline in Section IV the open problems concerning the symmetry properties of $\displaystyle\mathcal{D}$.

III Comparison with previous divergence functions

Given a general statistical manifold $\displaystyle(\mathrm{M},{\rm g},\nabla,\nabla^{*})$, the basic requirement for a smooth function $\displaystyle\mathcal{D}:\mathrm{M}\times\mathrm{M}\rightarrow\mathbb{R}$ to be a divergence on $\displaystyle\mathrm{M}$ is its consistency with the dual structure $\displaystyle({\rm g},\nabla,\nabla^{*})$ through Eqs. (5)-(6) and the positivity $\displaystyle\mathcal{D}(p,q)>0$ for all $\displaystyle p,q\in\mathrm{M}$ sufficiently close to each other such that $\displaystyle p\neq q$. The novel canonical divergence (17) succeeds to holding these properties (see Theorem 5 in Felice18 ).

In this section, we will show that the canonical divergence $\displaystyle\mathcal{D}$ can be interpreted as a generalization of the divergence $\displaystyle\mathrm{D}$ introduced by Ay and Amari. Indeed, we will see that these two divergences coincide on particular classes of statistical manifolds. In order to achieve this result, we investigate some geometric properties of the vector field $\displaystyle\Pi_{t}(p)$ given by Eq. (14) aiming to split such a vector field in terms of the difference vector $\displaystyle\mathrm{X}_{t}(p)$ given in Eq. (9). To be more precise, let us refer to Fig. 2 where the $\displaystyle\nabla$-geodesic $\displaystyle\sigma(t)\,(0\leq t\leq 1)$ connecting $\displaystyle\sigma(0)=p$ with $\displaystyle\sigma(1)=q$ is drawn. Then, for each $\displaystyle t\in[0,1]$ we can consider the $\displaystyle\nabla$-geodesic $\displaystyle\sigma_{t}(s)\,(0\leq s\leq 1)$ connecting $\displaystyle p$ with $\displaystyle\sigma(t)$ and the $\displaystyle\nabla^{*}$-geodesic $\displaystyle\sigma^{*}_{t}(s)\,(0\leq s\leq 1)$ connecting $\displaystyle p$ with $\displaystyle\sigma(t)$. The difference vector $\displaystyle\mathrm{X}_{t}(p)=\mathrm{X}(\sigma(t),p)$ at $\displaystyle\sigma(t)$ pointing to $\displaystyle p$ is given in terms of the inverse exponential map by $\displaystyle\mathrm{X}_{t}(p):=\exp^{-1}_{\sigma(t)}(p).$ Therefore, the opposite of $\displaystyle\mathrm{X}_{t}(p)$ can be viewed as the $\displaystyle\nabla$-parallel translation of $\displaystyle\mathrm{X}_{p}(t)=\exp_{p}^{-1}(\sigma(t))$ along the $\displaystyle\nabla$-geodesic $\displaystyle\sigma_{t}$, namely $\displaystyle-\mathrm{X}_{t}(p)={\rm P}_{\sigma_{t}}\mathrm{X}_{p}(t)\ .$ Consider now the loop $\displaystyle\Sigma_{t}$ based at $\displaystyle p$ and given by first traveling from $\displaystyle p$ to $\displaystyle\sigma(t)$ along the $\displaystyle\nabla^{*}$-geodesic $\displaystyle\sigma_{t}^{*}$ and then back from $\displaystyle\sigma(t)$ to $\displaystyle p$ along the reverse of the $\displaystyle\nabla$-geodesic $\displaystyle\sigma_{t}$. If $\displaystyle\Sigma_{t}$ lies in a sufficiently small neighborhood of $\displaystyle p$, then Felice18

[TABLE]

where

[TABLE]

with $\displaystyle\mathrm{X}^{*}(t)$ and $\displaystyle\mathrm{X}(t)$ being the $\displaystyle\nabla$-parallel transport of $\displaystyle\mathrm{X}_{p}^{*}(t)\,(:=\stackrel{{\scriptstyle*}}{{\exp_{p}}}^{-1}(\sigma(t)))$ and $\displaystyle\mathrm{X}_{p}(t)$, respectively, from $\displaystyle p$ to each point of $\displaystyle B_{t}$ along the unique $\displaystyle\nabla$-geodesic joining them. Here, $\displaystyle\mathcal{R}$ is the curvature tensor of $\displaystyle\nabla$, $\displaystyle B_{t}$ denotes the disc defined by the curve $\displaystyle\Sigma_{t}$ and $\displaystyle\mathrm{X}^{*}_{p}(t)$, $\displaystyle\mathrm{X}_{p}(t)$ are linearly independent. In addition, $\displaystyle{\rm P}$ within the integral denotes the $\displaystyle\nabla$-parallel translation from each point in $\displaystyle B_{t}$ to $\displaystyle p$ along the unique $\displaystyle\nabla$-geodesic segment joining them. Finally, by means of the property of the parallel transport, we obtain the following geometric relation between the vector $\displaystyle\Pi_{t}(p)$ and the opposite of the difference vector $\displaystyle\mathrm{X}_{t}(p)$ Felice18 ,

[TABLE]

By noticing that $\displaystyle{\rm P}_{\sigma_{t}}\mathrm{X}_{p}(t)=\dot{\sigma}_{t}(1)=t\ \dot{\sigma}(t)$ and inserting Eq. (19) into the definition (17) of $\displaystyle\mathcal{D}$, we obtain

[TABLE]

where $\displaystyle\mathrm{D}(p,q)$ is the divergence introduced in Ay15 and given by Eq. (10).

It is clear from Eq. (20) that particular conditions on the curvature tensor would lead to the required equivalence between $\displaystyle\mathcal{D}$ and $\displaystyle\mathrm{D}$. Actually, in Information Geometry classes of statistical manifolds are characterized by the conditions on the curvature tensors of $\displaystyle\nabla$ and $\displaystyle\nabla^{*}$ (see for instance Refs. Amari00 , Lauritzen87 , Zhang07 ). In the Table 1 we can see the categories of statistical manifolds on which the canonical divergence $\displaystyle\mathcal{D}$ reduces to the divergence $\displaystyle\mathrm{D}$ introduced in Ay15 . A statistical manifold $\displaystyle(\mathrm{M},{\rm g},\nabla,\nabla^{*})$ is self-dual when $\displaystyle\nabla=\nabla^{*}$. Therefore, in this case $\displaystyle\mathrm{M}$ becomes a Riemannian manifold endowed with the Levi-Civita connection. Hence, the vectors $\displaystyle\mathrm{X}_{p}(t)$ and $\displaystyle\mathrm{X}^{*}_{p}(t)$ coincide for all $\displaystyle t\in[0,1]$. Finally, the skew-symmetry of the curvature tensor $\displaystyle\mathcal{R}$ yields the property $\displaystyle\mathcal{R}(X,X)=0$ for any $\displaystyle X\in\mathcal{T}(\mathrm{M})$. When a manifold $\displaystyle\mathrm{M}$ is dually flat, it has a mutually dual affine coordinates $\displaystyle\{\boldsymbol{\theta}\},\,\{\boldsymbol{\eta}\}$ and two potentials $\displaystyle\{\psi,\varphi\}$ such that $\displaystyle\mathrm{D}(p,q)=\psi(\boldsymbol{\theta}_{p})-\varphi(\boldsymbol{\eta}_{q})+\boldsymbol{\theta}_{p}\cdot\boldsymbol{\eta}_{q}$Ay15 . This claims that the canonical divergence $\displaystyle\mathcal{D}$ coincides with the canonical divergence of Bregman type introduced in Amari00 by Amari and Nagaoka on dually flat manifolds. The concept of a symmetric statistical manifold, that is the information geometric analogue to a symmetric space in Riemannian geometry, was introduced in Henmi . Here, the authors employed the following conditions on the curvature tensor, $\displaystyle\nabla\mathcal{R}=0$ and $\displaystyle\mathrm{R}(Y,X,X,X):=\langle\mathcal{R}(Y,X)X,X\rangle=0$, in order to prove that their divergence function is independent of the particular path connecting any two points $\displaystyle p,q$ sufficiently close to each other. The connection between the canonical divergence $\displaystyle\mathcal{D}$ and the divergence introduced by Henmi and Kobayashi is widely discussed in Felice18 .

To summarize, Tab. 1 describes, from the top to the bottom, the statistical manifolds ordered from less generality to more generality where the equivalence between $\displaystyle\mathcal{D}$ and $\displaystyle\mathrm{D}$ is achieved. In this view, we can consider $\displaystyle\mathcal{D}$ as an extension of the divergence $\displaystyle\mathrm{D}$ to the very general statistical manifold $\displaystyle(\mathrm{M},{\rm g},\nabla,\nabla^{*})$.

Since Eq. (2) we know that in general a divergence function is not symmetric in its argument. However, the symmetry property owned by the canonical divergence of Bregman type on dually flat manifolds, namely $\displaystyle\mathrm{D}(q,p)=\mathrm{D}^{*}(p,q)$, shows the way for the further investigation about symmetry properties of $\displaystyle\mathcal{D}$ in the very general context of Information Geometry.

IV Future developments towards symmetry

The target of this section would be the description of the symmetry property $\displaystyle\mathcal{D}(q,p)=\mathcal{D}^{*}(p,q)$ for any statistical manifold $\displaystyle(\mathrm{M},{\rm g},\nabla,\nabla^{*})$. To this aim, we rely on the gradient–based approach to divergence which was introduced in Ay15 and further developed in Ay17 . This approach yields the following decompositions of $\displaystyle\Pi_{t}(p)$ and $\displaystyle\Pi_{t}^{*}(p)$ in terms of the canonical divergence gradient and its dual Felice18 ,

[TABLE]

where $\displaystyle\sigma_{t}(s)\,,\sigma_{t}^{*}(s)\,,(0\leq s\leq 1)$ are the $\displaystyle\nabla$ and $\displaystyle\nabla^{*}$ geodesics, respectively, from $\displaystyle p$ to $\displaystyle\gamma(t)$ for any arbitrary path $\displaystyle\gamma(t)$ connecting $\displaystyle p$ and $\displaystyle q$.

On the other hand, by means of the theory of minimum contrast geometry by Eguchi Eguchi92 , we can show that $\displaystyle\mathrm{grad}_{\gamma(t)}\mathcal{D}_{p}$ is parallel to the tangent vector of the $\displaystyle\nabla$-geodesic starting from $\displaystyle p$ and $\displaystyle\mathrm{grad}_{\gamma(t)}\mathcal{D}^{*}_{p}$ is parallel to the tangent vector of the $\displaystyle\nabla^{*}$-geodesic starting from $\displaystyle p$. This proves that Eqs. (21) and (22) supply orthogonal decompositions of $\displaystyle\Pi_{t}(p)$ and $\displaystyle\Pi_{t}^{*}(p)$, respectively. To see this, let us consider the level sets of $\displaystyle\mathcal{D}_{p}$ and $\displaystyle\mathcal{D}^{*}_{p}$:

[TABLE]

Then to each $\displaystyle q\in\mathcal{H}(\kappa)$ we can define the minimum contrast leaf of $\displaystyle\mathcal{D}$ at $\displaystyle q$ Eguchi85 :

[TABLE]

Let us now fix $\displaystyle q$. Since $\displaystyle q$ minimizes the set $\displaystyle\{\mathcal{D}(p,q)\,|\,p\in L_{q}\,,\,q\in\mathcal{H}\}$ it follows that the derivative of $\displaystyle\mathcal{D}(p,q)$ at $\displaystyle q$ along any direction $\displaystyle U$ tangent to $\displaystyle\mathcal{H}$ vanishes, namely

[TABLE]

where $\displaystyle\partial^{\prime}_{U}$ denotes the derivative at $\displaystyle q$ along the direction $\displaystyle U$. Thus we have that $\displaystyle\langle U,\Xi\rangle_{q}=0$ for all $\displaystyle U\in{\rm T}_{q}\mathcal{H}$ and $\displaystyle\Xi\in{\rm T}_{q}L_{q}$, or equivalently that the tangent space of $\displaystyle L_{q}$ coincides with the normal space of $\displaystyle\mathcal{H}$ at $\displaystyle q$ (see Fig. 3 for a cross-reference). In addition, by taking derivatives at $\displaystyle q$ along directions $\displaystyle\Xi\,,\Upsilon$ normal to $\displaystyle\mathcal{H}$ we have that Eguchi92

[TABLE]

where the first relation defines the second fundamental tensor $\displaystyle II$ with respect to the $\displaystyle\nabla$-connection. This implies that the second fundamental tensor with respect to $\displaystyle\nabla$ for $\displaystyle L_{q}$ vanishes at $\displaystyle q$. Therefore, according to the well-known Gauss formula Lee97

[TABLE]

we can see from Eq. (25) that the family of all curves which are orthogonal to the level set $\displaystyle\mathcal{H}$ are all $\displaystyle\nabla$-geodesics ending at $\displaystyle q$ (with a suitable choice of the parameter).

Analogously, we have that the family of all curves which are orthogonal to the level set $\displaystyle\mathcal{H}^{*}$ are all $\displaystyle\nabla^{*}$-geodesics ending at $\displaystyle q$ (with a suitable choice of the parameter).

In order to answer what the relation of $\displaystyle\mathcal{D}(p,q)$ and $\displaystyle\mathcal{D}(q,p)$ is, let us put $\displaystyle\widetilde{\mathcal{D}}(p,q):=\mathcal{D}(q,p)$. Then, by noticing that $\displaystyle-\partial_{i}\partial_{j}\partial_{k}^{\prime}\,\widetilde{\mathcal{D}}(p,q)=\Gamma^{*}_{ijk}$ and repeating the same arguments as above, we can show that $\displaystyle\mathrm{grad}_{q}\widetilde{\mathcal{D}}_{p}=c(q)\,\mathrm{grad}_{q}\mathcal{D}^{*}_{p}$, where $\displaystyle c:\mathrm{M}\rightarrow\mathbb{R}$ is a smooth function on $\displaystyle\mathrm{M}$. This proves that there exists a function $\displaystyle f:[0,\infty]\rightarrow\mathbb{R}^{+}$ and $\displaystyle f^{\prime}>0$ such that Felice18

[TABLE]

Though this relation holds for a very general statistical manifold $\displaystyle(\mathrm{M},{\rm g},\nabla,\nabla^{*})$, this result is still not satisfactory. However, in Tab. 2 we can see the classes of statistical manifolds where the relation $\displaystyle\mathcal{D}(q,p)=\mathcal{D}^{*}(p,q)$ holds. This occurs in dually flat manifolds analogously to the canonical divergence of Bregman type introduced in Amari00 . Moreover, the required symmetry also holds in the symmetric statistical manifolds, which constitutes a new result in the setting of Information Geometry Felice18 . Forthcoming investigation will address such a symmetry in the general case.