Projection Theorems of Divergences and Likelihood Maximization Methods

TL;DR

This paper explores projection theorems for various divergences used in robust statistics, linking them to likelihood maximization and sufficiency principles, and deriving new theorems for density power divergences.

Contribution

It establishes the equivalence between divergence projection methods and likelihood-based estimation, and derives a new projection theorem for density power divergences.

Findings

Projection theorems relate reverse and forward divergence projections.

Equivalence shown between divergence projections and estimating equations.

New projection theorem derived for density power divergences.

Abstract

Projection theorems of divergences enable us to find reverse projection of a divergence on a specific statistical model as a forward projection of the divergence on a different but rather "simpler" statistical model, which, in turn, results in solving a system of linear equations. Reverse projection of divergences are closely related to various estimation methods such as the maximum likelihood estimation or its variants in robust statistics. We consider projection theorems of three parametric families of divergences that are widely used in robust statistics, namely the R\'enyi divergences (or the Cressie-Reed power divergences), density power divergences, and the relative $\alpha$-entropy (or the logarithmic density power divergences). We explore these projection theorems from the usual likelihood maximization approach and from the principle of sufficiency. In particular, we show the equivalence of solving the estimation problems by the projection theorems of the respective divergences and by directly solving the corresponding estimating equations. We also derive the projection theorem for the density power divergences.

Equations

$$D_{\alpha }(P,Q):=\frac{1}{\alpha -1}\log \sum _{x\in \mathcal{X}}P(x{)}^{\alpha }Q(x{)}^{1-\alpha }.$$

$$B_{\alpha }(P,Q):=\frac{\alpha }{1-\alpha }\sum _{x\in \mathcal{X}}P(x)Q(x{)}^{\alpha -1}-\frac{1}{1-\alpha }\sum _{x\in \mathcal{X}}P(x{)}^{\alpha }+\sum _{x\in \mathcal{X}}Q(x{)}^{\alpha }.$$

$${\mathcal{I}}_{\alpha }(P,Q)$$

$$\underset{\alpha \to 1}{\lim }{B}_{\alpha }(P,Q)=\underset{\alpha \to 1}{\lim }{\mathcal{I}}_{\alpha }(P,Q)=\underset{\alpha \to 1}{\lim }{D}_{\alpha }(P,Q)=I(P,Q):=\sum _{x\in \mathcal{X}}P(x)\log \frac{P(x)}{Q(x)}.$$

$$\sum _{j=1}^{n}s(X_j;\theta )=0,$$

$$\sum _{x\in \mathcal{X}}\hat{P}(x)s(x;\theta )=0,$$

$$\sum _{x\in \mathcal{X}}\hat{P}(x{)}^{\alpha }{P}_{\theta }(x{)}^{1-\alpha }s(x;\theta )=0,$$

$$\sum _{x\in \mathcal{X}}\hat{P}(x)P_{\theta }(x{)}^{\alpha -1}s(x,\theta )=\sum _{x\in \mathcal{X}}{P}_{\theta }(x{)}^{\alpha }s(x,\theta ),$$

$$\frac{\sum _{x\in \mathcal{X}}\hat{P}(x)P_{\theta }(x{)}^{\alpha -1}s(x;\theta )}{\sum _{x\in \mathcal{X}}\hat{P}(x)P_{\theta }(x{)}^{\alpha -1}}=\frac{\sum _{x\in \mathcal{X}}{P}_{\theta }(x{)}^{\alpha }s(x;\theta )}{\sum _{x\in \mathcal{X}}{P}_{\theta }(x{)}^{\alpha }},$$

$$L_{1}^{(\alpha)}(\theta):=\dfrac{1}{1-\alpha}\log\Big{[}\sum\limits_{x\in\mathcal{X}}\widehat{P}(x)^{\alpha}P_{\theta}(x)^{1-\alpha}\Big{]},$$

$$L_2^{(\alpha )}(\theta ):=\frac{1}{n}\sum _{j=1}^n\left[\frac{\alpha P_{\theta }(X_j{)}^{\alpha -1}-1}{\alpha -1}\right]-\sum _{x\in \mathcal{X}}{P}_{\theta }(x{)}^{\alpha },$$

$$L_{3}^{(\alpha)}(\theta):=\dfrac{\alpha}{\alpha-1}\log\bigg{[}\dfrac{1}{n}\sum\limits_{j=1}^{n}P_{\theta}(X_{j})^{\alpha-1}\bigg{]}-\log\Big{[}\sum\limits_{x\in\mathcal{X}}P_{\theta}(x)^{\alpha}\Big{]}.$$

$$L(\theta ):=\frac{1}{n}\sum _{j=1}^n\log P_{\theta }(X_j).$$

$$\underset{P_{\theta }\in \Pi }{\min }D(\hat{P},P_{\theta }),$$

$$\underset{P\in \mathbb{C}}{\min }D(P,Q)$$

$$P_{\theta}(x)=Z(\theta)^{-1}\exp\big{[}\log(Q(x))+\theta^{T}f(x)\big{]}\quad\text{for }x\in\mathcal{X},$$

$${\mathbb{E}}_{\theta }[f(X)]=\bar{f},$$

$$\displaystyle P_{\theta}(x)=Z(\theta)^{-1}\big{[}Q(x)^{\alpha-1}+(1-\alpha)\theta^{T}f(x)\big{]}^{\frac{1}{\alpha-1}}\quad\text{for }~{}x\in\mathcal{X},$$

$$\displaystyle\mathbb{E}_{\theta}\big{[}f_{i}(X)\big{]}=\dfrac{\mathbb{E}_{\theta}\big{[}Q(X)^{\alpha-1}\big{]}}{\overline{{Q}^{\alpha-1}}}\bar{f_{i}},$$

$$\displaystyle P_{\theta}(x)=Z(\theta)^{-1}\big{[}Q(x)^{1-\alpha}+(1-\alpha)\theta^{T}f(x)\big{]}^{\frac{1}{1-\alpha}}\quad\text{for }~{}x\in\mathcal{X},$$

$$\sum _{x\in \mathcal{X}}{P}_{\theta }(x{)}^{\alpha }{f}_i(x)=\frac{\sum _{x\in \mathcal{X}}{P}_{\theta }(x{)}^{\alpha }Q(x{)}^{1-\alpha }}{\sum _{x\in \mathcal{X}}\hat{P}(x{)}^{\alpha }Q(x{)}^{1-\alpha }}\sum _{x\in \mathcal{X}}\hat{P}(x{)}^{\alpha }{f}_i(x),$$

$$\displaystyle\mathbb{E}_{\theta^{(\alpha)}}\big{[}f_{i}(X)\big{]}=\dfrac{\mathbb{E}_{\theta^{(\alpha)}}\big{[}Q(X)^{1-\alpha}\big{]}}{\overline{{Q}^{1-\alpha}}^{(\alpha)}}\overline{f_{i}}^{(\alpha)},$$

$$\displaystyle\Theta^{\prime}=\Big{\{}\theta^{\prime}=(\theta_{1}^{\prime},\dots,\theta_{k}^{\prime}):\theta^{\prime}_{i}=\tfrac{(-\alpha)\theta_{i}}{\|Q\|^{1-\alpha}},i=1,\dots,k,~{}\theta\in\Theta\Big{\}},$$

$$P_{\theta }^{(\alpha )}(x)$$

$$\displaystyle=\tfrac{Z(\theta)^{-\alpha}\|Q\|^{\alpha}}{\|P_{\theta}\|^{\alpha}}\Big{[}\Big{\{}{\tfrac{Q(x)^{\alpha}}{\|Q\|^{\alpha}}}\Big{\}}^{\frac{1-\alpha}{\alpha}}+(1-\tfrac{1}{\alpha})\frac{(-\alpha)}{\|Q\|^{1-\alpha}}\theta^{T}f(x)\Big{]}^{\frac{1}{\frac{1}{\alpha}-1}}$$

$$\displaystyle=\tfrac{Z(\theta)^{-\alpha}\|Q\|^{\alpha}}{\|P_{\theta}\|^{\alpha}}\Big{[}\big{\{}Q^{(\alpha)}(x)\big{\}}^{\frac{1}{\alpha}-1}+(1-\tfrac{1}{\alpha}){\theta^{\prime}}^{T}f(x)\Big{]}^{\frac{1}{\frac{1}{\alpha}-1}}.$$

$$P(x)$$

$$\displaystyle=Z(\theta^{\prime})^{-1}\Big{[}\Big{\{}\tfrac{Q(x)}{\|Q\|}\Big{\}}^{1-\alpha}+\tfrac{(1-\alpha)}{\|Q\|^{1-\alpha}}\theta^{T}f(x)\Big{]}^{\frac{\alpha}{1-\alpha}}$$

$$\displaystyle=\tfrac{Z(\theta^{\prime})^{-1}}{\|Q\|^{\alpha}}\big{[}Q(x)^{1-\alpha}+(1-\alpha)\theta^{T}f(x)\big{]}^{\frac{\alpha}{1-\alpha}}.$$

$$P(x)^{1/\alpha}=\tfrac{Z(\theta^{\prime})^{{-1/\alpha}}}{\|Q\|}\big{[}Q(x)^{1-\alpha}+(1-\alpha)\theta^{T}f(x)\big{]}^{\frac{1}{1-\alpha}},$$

$$P^{(1/\alpha )}(x)$$

Full text

** Projection theorems of divergences enable us to find reverse projection of a divergence on a specific statistical model as a forward projection of the divergence on a different but rather “simpler” statistical model, which, in turn, results in solving a system of linear equations. Reverse projection of divergences are closely related to various estimation methods such as the maximum likelihood estimation or its variants in robust statistics. We consider projection theorems of three parametric families of divergences that are widely used in robust statistics, namely the Rényi divergences (or the Cressie-Reed power divergences), density power divergences, and the relative $\alpha$-entropy (or the logarithmic density power divergences). We explore these projection theorems from the usual likelihood maximization approach and from the principle of sufficiency. In particular, we show the equivalence of solving the estimation problems by the projection theorems of the respective divergences and by directly solving the corresponding estimating equations. We also derive the projection theorem for the density power divergences.

Estimating equations, power divergence, power-law family, projection theorem, relative $\alpha$-entropy, Rényi divergence, robust estimation, sufficient statistics,

1 Introduction

Minimum divergence111By a divergence, we mean a non-negative extended real valued function defined for every pair of probability measures $(P,Q)$ satisfying $D(P,Q)=0$ if and only if $P=Q$. (or minimum distance) method has a unique place in statistical inference because of its ability to trade-off between efficiency and robustness [3, 28]. Minimization of Kullback-Leibler divergence ($I$-divergence) or relative entropy is closely related to the maximum likelihood estimation (MLE) [17, Lemma 3.1]. MLE is not a preferred method when the data set is contaminated by some unexpected sample points called outliers. However, $I$-divergence can be generalized, replacing the logarithmic function by some power function, to produce divergences that are more robust with respect to outliers [2, 21]. In this paper we consider three such families of divergences that are well-known in the context of robust statistics. They are defined as follows.

Let $\mathcal{X}$ be a finite set. Let $\mathcal{P}:=\mathcal{P}(\mathcal{X})$ be the space of all strictly positive probability measures222$\mathcal{P}$ can be just thought of the set of all ordered $|\mathcal{X}|$-tuples with strictly positive components that sum to one. on $\mathcal{X}$. Let $\alpha>0,\alpha\neq 1$, and let $P,Q\in\mathcal{P}$.

The $D_{\alpha}$-divergence333Upto a monotone function, same as the Cressie-Read power divergence [8]. (also known as Rényi divergence [30] or power divergence [29]):

[TABLE] 2. 2.

The $B_{\alpha}$-divergence (also known as density power divergence [2]):

[TABLE] 3. 3.

The $\mathscr{I}_{\alpha}$-divergence [31], [32], [26], [19] (also known as relative $\alpha$-entropy [23], [24], logarithmic density power divergence [27]):

[TABLE]

Throughout the paper $\log$ stands for the natural logarithm. It should be noted that, although the above divergences are not defined for $\alpha=1$, they all coincide with the $I$-divergence as $\alpha\rightarrow 1$ [7]. That is,

[TABLE]

In this sense each of these three classes of divergences can be regarded as a generalization of the $I$-divergence.

$D_{\alpha}$-divergences also arise as generalized cut-off rates in information theory [13]. The $B_{\alpha}$-divergences belong to the Bregman class which is characterized by transitive projection rules (see [12, Eq. (3.2) and Th. 3]). The $\mathscr{I}_{\alpha}$-divergence (for $\alpha<1$) arises in information theory as redundancy in mismatched guessing moments [31], in mismatched compression [24], and in mismatched encoding of tasks [5]. The three classes of divergences are associated with robust inference for $\alpha>1$ in case of $B_{\alpha}$ and $\mathscr{I}_{\alpha}$, and $\alpha<1$ in case of $D_{\alpha}$, as we shall see now.

Suppose that $X_{1},\dots,X_{n}$ are an independent and identically distributed (i.i.d.) random sample drawn according to a particular member of a parametric family of probability measures, $\Pi=\{P_{\theta}:\theta\in\Theta\}\subset\mathcal{P}$, where $\Theta$ is an open subset of $\mathbb{R}^{k}$. To find the MLE of the parameter $\theta$, one needs to solve the so-called score equation or estimating equation for $\theta$, given by

[TABLE]

where $s(x;\theta):=\nabla\log P_{\theta}(x)$, called the score function and $\nabla$ stands for gradient with respect to $\theta$. Observe that (1.5) can be re-written as

[TABLE]

where $\widehat{P}(\cdot)$ is the empirical probability measure of the sample $X_{1},\dots,X_{n}$.

In the presence of outliers in the observed sample, one modifies the score equation by scaling the score function in (1.6) by weights that down-weights the effect of outliers. The following estimating equation, referred as generalized Hellinger estimating equation, was proposed where the score function was weighted by $\widehat{P}(\cdot)^{\alpha}P_{\theta}(\cdot)^{1-\alpha}$ instead of $\widehat{P}(\cdot)$:

[TABLE]

where $\alpha\in(0,1)$. The above estimating equation was proposed based on the following intuition. If $x$ is an outlier in the sample, $\widehat{P}(x)^{\alpha}$ is larger while $P_{\theta}(x)^{1-\alpha}$ is smaller for sufficiently smaller value of $\alpha$. Hence the terms in (1.7) corresponding to outliers are down-weighted. (See [3, Sec. 4.3] and the references therein.) The following estimating equation, where the score function is weighted by power of model probability measure and equated to its hypothetical one, was proposed by Basu et. al. [2]:

[TABLE]

where $\alpha>1$. Motivated by the works of Field and Smith [18] and Windham [35], further an alternative estimating equation was proposed by Jones et. al. [21] where the weights in (1.8) were normalized:

[TABLE]

where $\alpha>1$. It should be noted that all the three estimating equations coincide with the usual score equation (1.6) when $\alpha=1$, since $\sum_{x}P_{\theta}(x)s(x;\theta)=0$. The estimating equations (1.6), (1.7), (1.8), and (1.9) are, respectively, associated with the divergences in (1.4), (1.1), (1.2), and (1.3) in a sense that will be made clear in the following.

Observe that the estimating equations (1.6), (1.7), (1.8), and (1.9) are implications of the first order optimality condition of, respectively, the usual log-likelihood function and the following modified likelihood functions,

[TABLE]

[TABLE]

and

[TABLE]

Although the above likelihood functions (1.10), (1.11) and (1.12) are not defined for $\alpha=1$, it can be shown that they all coincide with the following usual log-likelihood function as $\alpha\to 1$:

[TABLE]

Moreover, it is easy to see that the $P_{\theta}$ that maximizes the likelihood function in (1.10), (1.11), (1.12) or (1.13) is same as the one that minimizes $D_{\alpha}(\widehat{P},P_{\theta})$, $B_{\alpha}(\widehat{P},P_{\theta})$, $\mathscr{I}_{\alpha}(\widehat{P},P_{\theta})$ or $I(\widehat{P},P_{\theta})$ respectively. Thus, for MLE or “robustified MLE”, one needs to solve

[TABLE]

where $D$ is either $I$, $D_{\alpha}$, $B_{\alpha}$ or $\mathscr{I}_{\alpha}$. The minimizing probability measure $P_{\theta^{*}}$ (if exists and unique) is known as the reverse $D$-projection of $\widehat{P}$ on $\Pi$.

A “dual” minimization problem is the so-called forward projection problem, where the minimization is over the first argument of the divergence function. Given $\mathbb{C}\subset\mathcal{P}$, and $Q\in\mathcal{P}$, any $P^{*}\in\mathbb{C}$ that attains

[TABLE]

is called a forward $D$-projection of $Q$ on $\mathbb{C}$. Forward projection is usually on a convex set or on an $\alpha$-convex set of probability measures. Forward projection on a convex set is motivated by the well-known maximum entropy principle of statistical physics [20]. Motivation for forward projection on $\alpha$-convex set comes from the so-called non-extensive statistical physics [33, 34, 23]. Forward I-projection on convex sets was extensively studied by Csiszár [9, 10, 14], Csiszár and Matúš [16], [15], Csiszár and Shields [17], and Csiszár and Tusnády [11].

Csiszár and Shields showed that the reverse I-projection on an exponential family is same as the forward I-projection on a linear family444See Definition 2., which, in turn, is a solution to a system of linear equations [17, Th. 3.3]. We call such a result a projection theorem for the associated divergence. Projection theorem for I-divergence was due to an “orthogonal” relationship between the exponential family and the linear family. Projection theorem for $\mathscr{I}_{\alpha}$-divergence was established by Kumar and Sundaresan where the so-called $\alpha$-power-law family plays the role of exponential family [24, Th. 18 and Th. 21]. Projection theorem for $D_{\alpha}$-divergence was established by Kumar and Sason where a variant of the $\alpha$-power-law family, called an $\alpha$-exponential family, plays the role of exponential family and the so-called $\alpha$-linear family plays the role of linear family [25, Th. 6]. Projection theorem for the more general class of Bregman divergences, where $B_{\alpha}$ is a subclass, was established by Csiszár and Matúš [16] using techniques from convex analysis. One of our goals in this paper is to derive the projection theorem for the $B_{\alpha}$-divergence using elementary tools. We also identify the parametric family of probability measures associated with the projection theorem of $B_{\alpha}$-divergence, which is yet another generalization of the exponential family. In all these projection theorems, the Pythagorean theorem of the respective divergence plays a key role. Thus projection theorems enable us to find the estimator (whether MLE or robustified MLE) as a forward projection if the estimation is done on a specific parametric family. While for MLE the required family is the exponential family, for robustified MLE, it is one of the generalized exponential families.

Our contributions in this paper are the following.

• (a)

We show that the generalized Hellinger estimating equation (1.7) and the estimating equation of Jones et al. (1.9) are equivalent under a transformation.

• (b)

We show the equivalence of the two methods of solving the estimation problems: the one by the projection theorem and the other by directly solving the corresponding estimating equation.

• (c)

We show that the statistics of the data that have bearing on the projection theorem (that is, projection equation) are the sufficient statistics of the underlying statistical model with respect to the associated likelihood function.

• (d)

We derive the projection theorem for the $B_{\alpha}$-divergence.

Rest of the paper is organised as follows. The contributions mentioned in (a) and (b) are covered in section 2. The contribution mentioned in (c) is covered in section 3. The contribution in (d) is given in Appendix A. The paper ends with a summary in section 4.

2 Relationship Between Projection Theorems and Estimating Equations

We now show that the solutions of the estimation problems obtained using the projection theorems and those obtained by directly solving the estimating equations are the same. Recall that, projection theorem of a divergence enables us to find the reverse projection in terms of a forward projection if the former is done on the statistical model associated with the divergence. The statistical models associated with the $I$, $B_{\alpha}$, $\mathscr{I}_{\alpha}$ and $D_{\alpha}$ divergences are, respectively, exponential family, non-normalized $\alpha$-power-law family, $\alpha$-power-law family, and $\alpha$-exponential family which we shall define now. Indeed, for $I$, $B_{\alpha}$, $\mathscr{I}_{\alpha}$ and $D_{\alpha}$ divergences, the reverse projection (if exists) must satisfy an equation that we call a projection equation. We shall first list down the statistical models along with the projection equations associated with these divergences.

Projection equation for $I$-divergence on exponential family:

The exponential family, $\mathcal{E}:=\mathcal{E}(Q,f,\Theta)$, characterized by a probability measure $Q\in\mathcal{P}$, $k$ real valued functions $f_{i},i=1,\dots,k$ on $\mathcal{X}$, and parameter space $\Theta\subset\mathbb{R}^{k}$, is given by $\mathcal{E}=\{P_{\theta}:\theta\in\Theta\}\subset\mathcal{P}$, where

[TABLE]

$Z(\theta)$ is a normalizing constant that makes $P_{\theta}$ a probability measure, and $f=(f_{1},\dots,f_{k})^{T}$.

The reverse $I$-projection of $\widehat{P}$ on $\mathcal{E}$, if exists, is a unique solution of

[TABLE]

where $\bar{f}:=(\bar{f_{1}},\dots,\bar{f_{k}})^{T}$, $\bar{f_{i}}:=\frac{1}{n}\sum_{j=1}^{n}f_{i}(X_{j})$ for $i=1,\dots,k$, and $\mathbb{E}_{\theta}$ denotes expected value with respect to $P_{\theta}$. The above result was due to [17, Th. 3.3]. We call (2.2) the projection equation for $I$-divergence on exponential family. 2. 2.

Projection equation for $B_{\alpha}$ on non-normalized $\alpha$-power-law family:

The non-normalized $\alpha$-power-law family, $\mathbb{B}^{(\alpha)}:=\mathbb{B}^{(\alpha)}(Q,f,\Theta)\subset\mathcal{P}$, characterized by a $Q\in\mathcal{P}$, $k$ real valued functions $f_{i},i=1,\dots,k$ on $\mathcal{X}$, and parameter space $\Theta\subset\mathbb{R}^{k}$, is defined as in Definition 3.

The reverse $B_{\alpha}$-projection of $\widehat{P}$ on $\mathbb{B}^{(\alpha)}$, if exists, is a unique solution of

[TABLE]

by Theorem 7. (Notice that the projection equations for $I$ and $B_{\alpha}$-divergences are the same.) 3. 3.

Projection equation for $\mathscr{I}_{\alpha}$ on $\alpha$-power-law family:

The $\alpha$-power-law family, $\mathbb{M}^{(\alpha)}:=\mathbb{M}^{(\alpha)}(Q,f,\Theta)$555In the continuous case, each Student-t distribution can be seen as point-wise limit of a sequence of distributions from $\mathbb{M}^{(\alpha)}$. (c.f. [23, Rem. 13].), characterized by a $Q\in\mathcal{P}$ and $k$ real valued functions $f_{i},i=1,\dots,k$ on $\mathcal{X}$, and parameter space $\Theta\subset\mathbb{R}^{k}$, is given by $\mathbb{M}^{(\alpha)}=\{P_{\theta}:\theta\in\Theta\}\subset\mathcal{P}$, where

[TABLE]

and $Z(\theta)$ is the normalizing constant (c.f. [24, Defn. 8]).

The reverse $\mathscr{I}_{\alpha}$-projection of $\widehat{P}$ on $\mathbb{M}^{(\alpha)}$, if exists, is a unique solution of

[TABLE]

for $i=1,\dots,k$, where $\overline{{Q}^{\alpha-1}}:=\frac{1}{n}\sum_{j=1}^{n}Q(X_{j})^{\alpha-1}$ [24, Th. 18 and Th. 21]666Notice that this result in [24] is true under the assumption that the associated linear family $\mathbb{L}$ determined by the functions $f_{1},\dots,f_{k}$ is non-empty.. 4. 4.

Projection equation for $D_{\alpha}$ on $\alpha$-exponential family:

The $\alpha$-exponential family, $\mathscr{E}_{\alpha}:=\mathscr{E}_{\alpha}(Q,f,\Theta)$, characterized by a $Q\in\mathcal{P}$ and $k$ real valued functions $f_{i},i=1,\dots,k$ on $\mathcal{X}$, and parameter space $\Theta\subset\mathbb{R}^{k}$, is given by $\mathscr{E}_{\alpha}=\{P_{\theta}:\theta\in\Theta\}\subset\mathcal{P}$, where

[TABLE]

and $Z(\theta)$ is the normalizing constant (c.f. [25, Eq. (62)].

The reverse $D_{\alpha}$-projection of $\widehat{P}$ on $\mathscr{E}_{\alpha}$, if exists, is a unique solution of

[TABLE]

for $i=1,\dots,k$ ([25, Th. 6]).

(2.6) can be re-written as

[TABLE]

where $\mathbb{E}_{\theta^{(\alpha)}}$ denotes expectation with respect to $P_{\theta}^{(\alpha)}$; $\overline{{Q}^{1-\alpha}}^{(\alpha)}$ and $\overline{f_{i}}^{(\alpha)}$ are, respectively, averages of $Q(\cdot)^{1-\alpha}$ and $f_{i}(\cdot)$, with respect to $\widehat{P}^{(\alpha)}$, where $P^{(\alpha)}(x):=P(x)^{\alpha}/\sum_{y}P(y)^{\alpha}$ is the $\alpha$-scaled measure777Sometimes referred as escort measure among the physicists. associated with $P$.

Notice that while the projection equations for $I$ and $B_{\alpha}$ depend on the sample only through $\bar{f}$, the projection equations for $\mathscr{I}_{\alpha}$ and $D_{\alpha}$, respectively, depend on $\bar{f}/{\overline{{Q}^{\alpha-1}}}$ and $\overline{f}^{(\alpha)}/{\overline{{Q}^{1-\alpha}}^{(\alpha)}}$. This fact will be further explored in terms of sufficiency principle in section 3. Also notice that (2.4) and (2.7) are related by the transformations $P\leftrightarrow P^{(\alpha)}$ and $\alpha\leftrightarrow 1/\alpha$. The following lemmas explore this connection further. While lemma 1 establishes the connection between $\mathbb{M}^{(\alpha)}$ and $\mathscr{E}_{\alpha}$, lemma 2 establishes the connection between the estimation problem based on $D_{\alpha}$-divergence on $\mathscr{E}_{\alpha}$ and the estimation problem based on $\mathscr{I}_{\alpha}$-divergence on $\mathbb{M}^{(\alpha)}$. Lemma 1 is due to Karthik and Sundaresan [22, Th. 2], where only the reverse implication was proved though.

Lemma 1**.**

The map $P\mapsto P^{(\alpha)}$ establishes a one-to-one correspondence between the parametric families $\mathscr{E}_{\alpha}(Q,f,\Theta)$ and $\mathbb{M}^{(1/\alpha)}(Q^{(\alpha)},f,\Theta^{\prime})$, where

[TABLE]

and \|Q\|:=\big{[}\sum_{x}Q(x)^{\alpha}\big{]}^{1/\alpha}. (We suppress the dependence of $\|Q\|$ on $\alpha$ for notational convenience.)

Proof.

For any $P_{\theta}\in\mathscr{E}_{\alpha}(Q,f,\Theta)$, we have from (2.5), for $x\in\mathcal{X}$,

[TABLE]

Hence $P_{\theta}^{(\alpha)}\in\mathbb{M}^{(1/\alpha)}(Q^{(\alpha)},f,\Theta^{\prime})$. So, the mapping is well-defined. The map is clearly one-one, since it is easy to see that, if $P_{\theta}^{(\alpha)}=P_{\eta}^{(\alpha)}$ for some $\theta,\eta\in\Theta$, then $P_{\theta}=P_{\eta}$. To verify it is onto, let $P\in\mathbb{M}^{(1/\alpha)}(Q^{(\alpha)},f,\Theta^{\prime})$ be arbitrary. Then

[TABLE]

This implies that

[TABLE]

and hence

[TABLE]

Hence $P^{(1/\alpha)}\in\mathscr{E}_{\alpha}(Q,f,\Theta)$ and so $P^{(1/\alpha)}=P_{\theta}$ for some $\theta\in\Theta$. It is now easy to show that $P_{\theta}^{(\alpha)}=P$. Thus, for any $P\in\mathbb{M}^{(\tiny{1/\alpha})}(Q^{(\alpha)},f,\Theta^{\prime})$, there exists $P_{\theta}\in\mathscr{E}_{\alpha}(Q,f,\Theta)$ such that $P^{(\alpha)}_{\theta}=P$. Hence the mapping is onto.

∎

Lemma 2**.**

The following statements hold.

1. (a)

Solving the estimating equation (1.7) for $P_{\theta}\in\mathscr{E}_{\alpha}(Q,f,\Theta)$ is equivalent to solving the estimating equation (1.9) for the $\alpha$-scaled measure $P_{\theta}^{(\alpha)}\in\mathbb{M}^{(\tiny{1/\alpha})}(Q^{(\alpha)},f,\Theta^{\prime})$, where $\Theta^{\prime}$ is as in (2.8). 2. (b)

Solving the projection equation (2.7) for $P_{\theta}\in\mathscr{E}_{\alpha}(Q,f,\Theta)$ is equivalent to solving the projection equation (2.4) for the $\alpha$-scaled measure $P_{\theta}^{(\alpha)}\in\mathbb{M}^{({1/\alpha})}(Q^{(\alpha)},f,\Theta^{\prime})$, where $\Theta^{\prime}$ is as in (2.8).

Proof.

• (a)

The estimating equation in (1.7) can be re-written as

[TABLE]

since $\sum\limits_{x\in\mathcal{X}}P_{\theta}(x)s(x;\theta)=0$. This can further be re-written as

[TABLE]

Observe that

[TABLE]

Hence

[TABLE]

where $A(\theta)=\nabla\log\|P_{\theta}\|$. Plugging (2.12) in (2.11), we get

[TABLE]

This is same as (1.9) with $\widehat{P}$, $P_{\theta}$, and $\alpha$, respectively, replaced by $\widehat{P}^{(\alpha)}$, $P_{\theta}^{(\alpha)}$, and $1/\alpha$. Thus, solving the estimating equation (1.7) for $P_{\theta}\in\mathscr{E}_{\alpha}(Q,f,\Theta)$ is equivalent to solving (1.9) for $P_{\theta}^{(\alpha)}\in\mathbb{M}^{({1/\alpha})}(Q^{(\alpha)},f,\Theta^{\prime})$, by lemma 1.

• (b)

The projection equation (2.7) can further be re-written as

[TABLE]

This is same as (2.4) with $P_{\theta}$, $\widehat{P}$, $Q$, and $\alpha$, respectively, replaced by $P_{\theta}^{(\alpha)}$, $\widehat{P}^{(\alpha)}$, $Q^{(\alpha)}$, and $1/\alpha$. Thus solving (2.6) for $P_{\theta}\in\mathscr{E}_{\alpha}(Q,f,\Theta)$ is equivalent to solving (2.14) for $P_{\theta}^{(\alpha)}\in\mathbb{M}^{(1/\alpha)}(Q^{(\alpha)},f,\Theta^{\prime})$, by lemma 1.

∎

We are now ready to state the main result of this section.

Theorem 1**.**

The solutions of the estimation problems based on the estimating equations (1.6), (1.7), (1.8), and (1.9) on $\mathcal{E}$, $\mathscr{E}_{\alpha}$, $\mathbb{B}^{(\alpha)}$, and $\mathbb{M}^{(\alpha)}$ respectively, is same as the solutions obtained from their corresponding projection equations.

Proof.

1. (a)

MLE on exponential family $\mathcal{E}$:

For $P_{\theta}\in\mathcal{E}$,

[TABLE]

Hence (1.5) implies that

[TABLE]

Since $\mathbb{E}_{\theta}[\nabla\log P_{\theta}(X)]=0$, (2.15) implies that

[TABLE]

Thus, from (2.16) and (2.17), we see that the MLE must satisfy

[TABLE]

which is same as the projection equation for $I$-divergence. 2. (b)

Robust estimation based on (1.8) on $\mathbb{B}^{(\alpha)}$:

The estimating equation (1.8) can be re-written as

[TABLE]

For $P_{\theta}\in\mathbb{B}^{(\alpha)}$, from (A.22) we have

[TABLE]

Substituting this in (2.18), we get

[TABLE]

which is same as the projection equation for $B_{\alpha}$-divergence. 3. (c)

Robust estimation based on (1.9) on $\mathbb{M}^{(\alpha)}$:

The estimating equation (1.9) can be re-written as

[TABLE]

Now if $P_{\theta}\in\mathbb{M}^{(\alpha)}$, then from (2.3) we have

[TABLE]

Differentiating (2.20) with respect to $\theta$, we get

[TABLE]

Using (2.20) and (2.21), left-hand side of (2.19) becomes

[TABLE]

Similarly, using (2.20) and (2.21), one can show that the right-hand side of (2.19) is

[TABLE]

Hence (2.19) is same as

[TABLE]

Using (2.20) the above can be re-written as

[TABLE]

Hence the estimator $P_{\theta}$ must satisfy the above equation.

On the other hand, the projection equation (2.4) can be re-written as

[TABLE]

This implies

[TABLE]

The above two equations can be combined to

[TABLE]

This implies that the reverse projection $P_{\theta}$ must satisfy

[TABLE]

This is same as (2.22) derived from the estimating equation. Hence any solution of the projection equation (2.4) is a solution of the estimating equation (2.22). Hence, it suffices to show that, if (2.22) has a solution, it is unique. To this end, we define $\Phi:=(\phi_{1},\dots,\phi_{k})$, where

[TABLE]

where the second equality follows from (2.20). Then (2.22) reduces to solving

[TABLE]

for $\theta$. Thus it remains to show that the mapping $\theta\mapsto\Phi(\theta)$ is one-one. This is indeed the case. The proof is given in Appendix B. 4. (d)

Robust estimation based on (1.7) on $\mathscr{E}_{\alpha}$:

This clearly follows from (c) in view of lemmas 1 and 2.

∎

3 Projection Theorems and the Principle of Sufficiency

In the previous section we studied four estimation problems respectively on four statistical models from the perspectives of projection theorems and estimating equations. We observed that the estimating equations or the projection equations arising from these estimation problems depended on the given sample only through some specific statistics of the sample. In this section we explore this observation further from the principle of sufficiency. The following result, known as factorization theorem [6, Th. 6.2.6], tells us a way to identify the sufficient statistics from the log-likelihood function.

Factorization Theorem: Let $X_{1},\dots,X_{n}$ be an i.i.d. random sample drawn according to some model $P_{\theta},\theta\in\Theta$. A statistic $T(X_{1},\dots,X_{n})$ is a sufficient statistic for $\theta$ if and only if there exists functions $g(\theta,T(X_{1},\dots,X_{n}))$ and $h(X_{1},\dots,X_{n})$ such that the log-likelihood function $L(\theta)$ in (1.13) can be written as

[TABLE]

for all sample points $(X_{1},\dots,X_{n})$ and for all $\theta\in\Theta$.

However, when the data set is contaminated, we saw in section 1 that one should use some modified likelihood function for inference instead of the usual log-likelihood function. The quantities in (1.10), (1.11) and (1.12) are some examples of modified likelihood functions. Suppose that $L_{G}$ is one of the modified likelihood functions in (1.10) - (1.12). The robust estimator of $\theta$ is given by

[TABLE]

Now, if we can analogously write the modified likelihood function $L_{G}$ as in (3.1), that is, if

[TABLE]

for some functions $g$ and $h$, then we have

[TABLE]

This means that the robustified MLE depends on the sample only through the function $T(\cdot)$. Thus it is reasonable to call such $T(\cdot)$ a sufficient statistics for $\theta$ with respect to the modified likelihood function. In view of this, in the following theorem, we find the sufficient statistics for the parameters of each of the families $\mathcal{E}$, $\mathbb{B}^{(\alpha)}$, $\mathbb{M}^{(\alpha)}$ and $\mathscr{E}_{\alpha}$ when the appropriate likelihood function is used.

Theorem 2**.**

Let $X_{1},\dots,X_{n}$ be an i.i.d. random sample drawn according to one of the statistical models $\mathcal{E}$, $\mathbb{B}^{(\alpha)}$, $\mathbb{M}^{(\alpha)}$ or $\mathscr{E}_{\alpha}$. Then the following statements hold.

• (a)

For $\mathcal{E}$, $T_{1}(X_{1},\dots,X_{n})=\bar{f}$ is a sufficient statistic for $\theta$ with respect to the usual log-likelihood function $L(\theta)$ in (1.13).

• (b)

For $\mathbb{B}^{(\alpha)}$, $T_{2}(X_{1}\dots,X_{n})=\bar{f}$ is a sufficient statistic for $\theta$ with respect to the likelihood function $L_{2}^{(\alpha)}(\theta)$ in (1.11).

• (c)

For $\mathbb{M}^{(\alpha)}$, $T_{3}(X_{1}\dots,X_{n})={\bar{f}}/{\overline{Q^{\alpha-1}}}$ is a sufficient statistic for $\theta$ with respect to the likelihood function $L_{3}^{(\alpha)}(\theta)$ in (1.12).

• (d)

For $\mathscr{E}_{\alpha}$, $T_{4}(X_{1}\dots,X_{n})={{\overline{f}}^{(\alpha)}}/{{\overline{Q^{1-\alpha}}}^{(\alpha)}}$ is a sufficient statistic for $\theta$ with respect to the likelihood function $L_{1}^{(\alpha)}(\theta)$ in (1.10).

Proof.

• (a)

See [6, Th. 6.2.10].

• (b)

For $P_{\theta}\in\mathbb{B}^{(\alpha)}$, using (A.22), the likelihood function in (1.11) can be re-written as

[TABLE]

where

$h(X_{1},\dots,X_{n}):=\tfrac{\alpha}{\alpha-1}\overline{Q^{\alpha-1}}$,

$T_{2}(X_{1},\dots,X_{n}):=\bar{f}$,

and g(\theta,T_{2}(X_{1},\dots,X_{n})):=-\Big{[}\alpha Z(\theta)+\frac{1}{\alpha-1}+\alpha\theta^{T}\bar{f}+\sum\limits_{x\in\mathcal{X}}P_{\theta}(x)^{\alpha}\Big{]}.

Hence $T_{2}(X_{1},\dots,X_{n})=\bar{f}$ is a sufficient statistics for $\theta$.

• (c)

For $P_{\theta}\in\mathbb{M}^{(\alpha)}$, using (2.20), the likelihood function in (1.12) can be re-written as

[TABLE]

where $T_{3}(X_{1},\dots,X_{n}):={\bar{f}}/{\overline{Q^{\alpha-1}}}$, $h(X_{1},\dots,X_{n}):=\frac{\alpha}{\alpha-1}\log\overline{Q^{\alpha-1}}$, and

[TABLE]

Hence $T_{3}(X_{1},\dots,X_{n})={\bar{f}}/{\overline{Q^{\alpha-1}}}$ is a sufficient statistics for $\theta$.

• (d)

For $P_{\theta}\in\mathscr{E}_{\alpha}$, using (2.5), the likelihood function in (1.10) can be re-written as

[TABLE]

where

$T_{4}(X_{1},\dots,X_{n}):={{\overline{f}}^{(\alpha)}}/{{\overline{Q^{1-\alpha}}}^{(\alpha)}}$ with ${\overline{f}}^{(\alpha)}$ and ${\overline{Q^{1-\alpha}}}^{(\alpha)}$ being, respectively, averages of $f(\cdot)$ and $Q(\cdot)^{1-\alpha}$ with respect to the measure $\widehat{P}^{(\alpha)}$,

g(\theta,T_{4}(X_{1},\dots,X_{n})):=-\log Z(\theta)+\dfrac{1}{1-\alpha}\log\Big{[}1+(1-\alpha)\left\{{\theta^{T}{\overline{f}}^{(\alpha)}}/{{\overline{Q^{1-\alpha}}}^{(\alpha)}}\right\}\Big{]}, and

$h(X_{1},\dots,X_{n}):=\tfrac{1}{1-\alpha}\log{\sum\limits_{x\in\mathcal{X}}\widehat{P}(x)^{\alpha}Q(x)^{1-\alpha}}$.

Hence $T_{4}(X_{1},\dots,X_{n})={{\overline{f}}^{(\alpha)}}/{{\overline{Q^{1-\alpha}}}^{(\alpha)}}$ is a sufficient statistic for $\theta$.

∎

Thus the sufficient statistics $\bar{f}$, ${\bar{f}}/{\overline{Q^{\alpha-1}}}$, and ${{\overline{f}}^{(\alpha)}}/{{\overline{Q^{1-\alpha}}}^{(\alpha)}}$ are precisely the statistics of the sample that influence the projection equations (2.2), (2.4), and (2.7) respectively.

4 Summary

In this paper we studied four estimation problems on four statistical models distinctively associated with them. Each of these estimation problems can be regarded as a reverse projection problem of one of the divergences $I$, $B_{\alpha}$, $\mathscr{I}_{\alpha}$ and $D_{\alpha}$, on one of these statistical models. Projection theorems tell us that, the reverse projection, if exists, is the unique solution of an equation, called projection equation. While the projection equations are simple and easy to interpret in the case of $I$ and $B_{\alpha}$-divergences, it is not so in the case of $\mathscr{I}_{\alpha}$ and $D_{\alpha}$-divergences. Our objective in this paper was to understand these projection theorems (that is, projection equations) from the usual statistical method of solving by estimating equations. In the case of $I$ and $B_{\alpha}$-divergences, we saw that the projection equations and the estimating equations are the same and both depended only on the statistics of $f$ which is one of the defining entity of the underlying statistical model. However, in the case of $\mathscr{I}_{\alpha}$ and $D_{\alpha}$-divergences, the estimating equations and the projection equations are different and both depended not only on the statistics of $f$ but also on $Q$ which is other defining entity of the model. Nevertheless, we showed that, in the case of $\mathscr{I}_{\alpha}$ and $D_{\alpha}$ also, the solutions of the estimation problems obtained both from the projection equations and from the estimating equations are the same. We then tried to understand the projection theorems via sufficient statistics in the sense of generalized likelihood functions and observed that the estimating equations (or the projection equations) depended on the given sample only through these sufficient statistics.

Appendix A Projection Theorem for Density Power Divergence

The Projection theorem and the Pythagorean property of the more general class of Bregman divergences were established by Csiszár and Matúš [16] in generality using tools from convex analysis. The density power divergences $B_{\alpha}$ is a subclass of the Bregman divergences. However, it is hard to extract the results for the $B_{\alpha}$-divergence from [16]. Hence we derive those results for the $B_{\alpha}$-divergence using some elementary tools as in [17] for the KL-divergence. We must point out that the geometry of $B_{\alpha}$-divergence is quite a natural extension of that of KL-divergence. Here we assume that $\mathcal{P}:=\mathcal{P}(\mathcal{X})$ is the space of all probability measures on $\mathcal{X}$. Let $B_{\alpha}$ be as defined in (1.2)888We assume the usual convention that $B_{\alpha}(P,Q)=\infty$ when $P\not\ll Q$ and $\alpha<1$.. Let us also recall the definitions of reverse and forward projections given in (1.14) and (1.15).

Definition 1**.**

For $P\in\mathcal{P}$, the support of $P$ is defined as $\text{Supp}(P)=\{x\in\mathcal{X}:P(x)>0\}$. For $\mathbb{C}\subset\mathcal{P}$, we will denote $\text{Supp}(\mathbb{C})$ for the union of support of members of $\mathbb{C}$.

First we show that, while the Pythagorean inequality is always satisfied when the forward projection is on a closed convex set, equality is satisfied when the forward projection is on a linear family and $\alpha<1$. Here, in the sequel, we assume that $\text{Supp}(Q)=\mathcal{X}$.

Theorem 3**.**

Let $\alpha>0,\alpha\neq 1$. Let $P^{*}$ be the forward $B_{\alpha}$-projection of $Q$ on a closed and convex set $\mathbb{C}$. Then

[TABLE]

Further, if $\alpha<1$, $\text{Supp}(\mathbb{C})=\text{Supp}(P^{*})$.

Proof.

Let $P\in\mathbb{C}$ and define

$P_{t}(\cdot)=(1-t)P^{*}(\cdot)+tP(\cdot)$, for $t\in[0,1]$.

Since $\mathbb{C}$ is convex, $P_{t}\in\mathbb{C}$. By mean-value theorem, we have for each $t\in(0,1)$,

[TABLE]

Using definition of $B_{\alpha}$, we have

[TABLE]

Therefore, using (A.2),

[TABLE]

Hence, as $t\downarrow 0$, we have

[TABLE]

which is equivalent to (A.1).

If $\text{Supp}(P^{*})\neq\text{Supp}(\mathbb{C})$, that is, if $P^{*}(x)=0$ for some $x\in\mathcal{X}$ and some $P\in\mathbb{C}$ such that $x\in\text{Supp}(P)$, and if $\alpha<1$, then the left-hand side of (A.3) goes to $-\infty$ as $t\downarrow 0$, which contradicts (A.3). This proves the claim. ∎

If $\alpha>1$, in general, $\text{Supp}(P^{*})\neq\text{Supp}(\mathbb{C})$. [24, Example 2] serves as a counterexample here as well.

We will now show that equality holds in (A.1) for $\alpha<1$ when $\mathbb{C}$ is a linear family, which we shall define now.

Definition 2**.**

The linear family, determined by $k$ real valued functions $f_{i},i=1,\dots,k$ on $\mathcal{X}$ and $k$ real numbers $a_{i},i=1,\dots,k$, is defined as

[TABLE]

Theorem 4**.**

Let $P^{*}$ be the forward $B_{\alpha}$-projection of $Q$ on $\mathbb{L}$. The following hold.

1. (a)

If $\alpha<1$, then the Pythagorean equality holds, that is,

[TABLE] 2. (b)

If $\alpha>1$ and if $\text{Supp}(P^{*})=\text{Supp}(\mathbb{L})$, then the Pythagorean equality (A.6) holds.

Proof.

(a) Let $P_{t}$ be as in Theorem 3. Since $\text{Supp}(P^{*})=\text{Supp}(\mathbb{L})$, there exists $t^{\prime}<0$ such that $P_{t}=(1-t)P^{*}+tP\in\mathbb{L}$ for $t\in(t^{\prime},0)$. Hence, for every $t\in(t^{\prime},0)$, there exists $\tilde{t}\in(t,0)$ such that

[TABLE]

Proceeding as in Theorem 3, we get (A.4) with a reversed inequality. Thus we have equality in (A.4). Hence (A.6) holds.

(b) Similar to (a). ∎

Remark 1**.**

When $\alpha>1$, equality in (A.6) does not hold in general. [24, Example 2] serves as a counterexample here as well.

We will now find an explicit expression of the forward $B_{\alpha}$-projection in both the cases $\alpha<1$ and $\alpha>1$ separately.

Theorem 5**.**

Let $Q\in\mathcal{P}$ and consider a linear family $\mathbb{L}$ of probability measures as in (A.5).

1. (a)

If $\alpha<1$, the forward $B_{\alpha}$-projection $P^{*}$ of $Q$ on $\mathbb{L}$ satisfies

[TABLE]

where $\theta_{1},\dots,\theta_{k}$ are scalars and $Z$ is a constant. 2. (b)

If $\alpha>1$, the forward $B_{\alpha}$-projection $P^{*}$ of $Q$ on $\mathbb{L}$ satisfies

[TABLE]

where $\theta_{1},\dots,\theta_{k}$ are scalars, $Z$ is a constant, and for any real number $r$, $[r]_{+}:=\max\{r,0\}$.

Proof.

(a) The linear family in (A.5) can be re-written as

[TABLE]

Let $\mathcal{F}$ be the subspace of $\mathbb{R}^{|\text{Supp}(\mathbb{L})|}$ spanned by the $k$ vectors $f_{1}(\cdot)-a_{1},\dots,f_{k}(\cdot)-a_{k}$. Then every $P\in\mathbb{L}$ can be thought of a $|\text{Supp}(\mathbb{L})|$-dimensional vector in $\mathcal{F}^{\perp}$. Hence $\mathcal{F}^{\perp}$ is a subspace of $\mathbb{R}^{|\text{Supp}(\mathbb{L})|}$ that contains a vector whose components are strictly positive as $P^{*}\in\mathbb{L}$. It follows that $\mathcal{F}^{\perp}$ is spanned by its probability vectors. From (A.4), we see that (A.6) is equivalent to

[TABLE]

This implies that the vector

[TABLE]

Hence

[TABLE]

for some scalars $c_{1},\dots,c_{k}$. This implies (A.7) for appropriate choices of $Z$ and $\theta_{1},\dots,\theta_{k}$.

(b) The proof of this case is similar to the proof for relative $\alpha$-entropy [24, Th. 14 (b)]. The optimization problem involved in the forward $B_{\alpha}$-projection is

[TABLE]

Hence, by [4, Prop. 3.3.7], there exists Lagrange multipliers $\lambda_{1},\dots,\lambda_{k}$, $\nu$, and $(\mu(x),x\in\mathcal{X})$ respectively associated with the above constraints such that, for $x\in\mathcal{X}$,

[TABLE]

Since

[TABLE]

(A.15) can be re-written as

[TABLE]

Multiplying both sides by $P^{*}(x)$ and summing over all $x\in\mathcal{X}$, we get

[TABLE]

For $x\in\text{Supp}(P^{*})$, from (A.17), we must have $\mu(x)=0$. Then, from (A.19), we have

[TABLE]

If $P^{*}(x)=0$, from (A.19), we get

[TABLE]

Combining (A.20) and (A.21) we get (A.8). ∎

Remark 2**.**

The expression for $P^{*}$ in (A.7) (or in (A.8)) is given with the multiplicative factor $1-\alpha$ in the square brackets so that the formula in (A.7) can be re-written, with the help of $\alpha$-exponential and $\alpha$-logarithmic functions [24, Defn. 7], as P^{*}(x)^{-1}=e_{\alpha}\big{[}\ln_{\alpha}(Q(x)^{-1})+Z+\small\sum\limits_{i=1}^{k}\theta_{i}f_{i}(x)\big{]}, analogous to the exponential probability measure (see (2.1)).

Theorem 5 suggests us to define a parametric family of probability measures that extends the usual exponential family. We first formally define this family and then show an orthogonality relationship between this family and the linear family. As a consequence we will also show that the reverse $B_{\alpha}$-projection on this generalized exponential family is same as a forward projection on a linear family.

Definition 3**.**

The non-normalized $\alpha$-power-law family999This term is coined due to the fact that this family differs from the $\alpha$-power-law family only in the normalizing constant., $\mathbb{B}^{(\alpha)}:=\mathbb{B}^{(\alpha)}(Q,f,\Theta)$, characterized by a $Q\in\mathcal{P}$, $k$ real valued functions $f_{i},i=1,\dots,k$ on $\mathcal{X}$, and a parameter space $\Theta\subset\mathbb{R}^{k}$, is given by $\mathbb{B}^{(\alpha)}=\{P_{\theta}:\theta\in\Theta\}\subset\mathcal{P}$, where

[TABLE]

Remark 3**.**

• (a)

Observe that $\mathbb{B}^{(\alpha)}$ is a special case of the family $\mathcal{F}_{[\beta h]}$ in **[16, Eq. (28)]** when $h(\cdot)=Q(\cdot)$ and $\beta(\cdot,t)=\frac{1}{\alpha-1}[t^{\alpha}-\alpha t+\alpha-t]$.

• (b)

As in the case of $\mathcal{E}$, $\mathbb{M}^{(\alpha)}$, and $\mathscr{E}_{\alpha}$ families, the family $\mathbb{B}^{(\alpha)}$ depends on the reference measure $Q$ only in a loose manner in the sense that any other member of $\mathbb{B}^{(\alpha)}$ can play the role of $Q$. The change of reference measure only corresponds to a translation of the parameter space.

The following theorem and its corollary together establish an “orthogonality” relationship between the non-normalized $\alpha$-power-law family and its associated linear family.

Theorem 6**.**

Let $\alpha<1$. Let $\mathbb{B}^{(\alpha)}$ be the non-normalized $\alpha$-power-law family as in Definition 3 and $\mathbb{L}$ be the corresponding linear family determined by the same functions $f_{i},i=1,\dots,k$ and some constants $a_{i},i=1,\dots,k$ as in (A.5). If $P^{*}$ is the forward $B_{\alpha}$-projection of $Q$ on $\mathbb{L}$, then we have the following:

1. (a)

$\mathbb{L}\cap\text{cl}(\mathbb{B}^{(\alpha)})=\{P^{*}\}$* and*

[TABLE] 2. (b)

Further, if $\text{Supp}(\mathbb{L})=\mathcal{X}$, then $\mathbb{L}\cap\mathbb{B}^{(\alpha)}=\{P^{*}\}$.

Proof.

By Theorem 5, the forward $B_{\alpha}$-projection $P^{*}$ of $Q$ on $\mathbb{L}$ is in $\mathbb{B}^{(\alpha)}$. This implies that $P^{*}\in\mathbb{L}\cap\mathbb{B}^{(\alpha)}$. In general, it would suffice if we prove the following:

1. (i)

Every $\tilde{P}\in\mathbb{L}\cap\text{cl}(\mathbb{B}^{(\alpha)})$ satisfies (A.6) with $\tilde{P}$ in place of $P^{*}$. 2. (ii)

$\mathbb{L}\cap\text{cl}(\mathbb{B}^{(\alpha)})$ is non-empty.

We now proceed to prove both (i) and (ii).

(i) Let $\tilde{P}\in\mathbb{L}\cap\text{cl}(\mathbb{B}^{(\alpha)})$. As $\tilde{P}\in\text{cl}(\mathbb{B}^{(\alpha)})$, this implies there exists a sequence $\{P_{n}\}\subset\mathbb{B}^{(\alpha)}$ such that $P_{n}\rightarrow\tilde{P}$ as $n\to\infty$. Since $P_{n}\in\mathbb{B}^{(\alpha)}$, we can write

[TABLE]

for some constants $\theta^{(n)}=(\theta_{1}^{(n)},\dots,\theta_{k}^{(n)})\in\mathbb{R}^{k}$ and $Z_{n}$. Now, for any $P\in\mathbb{L}$ we have, from the definition of linear family, $\sum\limits_{x\in\mathcal{X}}P(x)f_{i}(x)=a_{i},i=1,\dots,k$. Since $\tilde{P}\in\mathbb{L}$, we also have $\sum\limits_{x\in\mathcal{X}}\tilde{P}(x)f_{i}(x)=a_{i},i=1,\dots,k$. Multiplying both sides of (A.24) by $P(\cdot)$ and $\tilde{P}(\cdot)$ separately, we get

[TABLE]

and

[TABLE]

which imply

[TABLE]

As $n\to\infty$, the above becomes

[TABLE]

which is equivalent to (A.6).

(ii) Let $P_{n}^{*}$ be the forward $B_{\alpha}$-projection of $Q$ onto the linear family

[TABLE]

By construction, $\left(1-\frac{1}{n}\right)P+\frac{1}{n}Q\in\mathbb{L}_{n}$ for any $P\in\mathbb{L}$. Hence, since $\text{Supp}(Q)=\mathcal{X}$, we have $\text{Supp}(\mathbb{L}_{n})=\mathcal{X}$. Since $\mathbb{L}_{n}$ is also characterized by the same functions $f_{i},i=1,\dots,k$, we have $P^{*}_{n}\in\mathbb{B}^{(\alpha)}$ for every $n\in\mathbb{N}$. Hence limit of any convergent subsequence of $\{P^{*}_{n}\}$ belongs to $\text{cl}(\mathbb{B}^{(\alpha)})\cap\mathbb{L}$. Thus $\text{cl}(\mathbb{B}^{(\alpha)})\cap\mathbb{L}$ is non-empty. This completes the proof. ∎

Corollary 1**.**

Let $\mathbb{L}$ and $\mathbb{B}^{(\alpha)}$ be characterized by the same functions $f_{i},i=1,\dots,k$. Then $\mathbb{L}\cap\text{cl}(\mathbb{B}^{(\alpha)})=\{P^{*}\}$ and

[TABLE]

Proof.

By Theorem 6, we have $\mathbb{L}\cap\text{cl}(\mathbb{B}^{(\alpha)})=\{P^{*}\}$. In view of Remark 3(b), notice that every member of $\mathbb{B}^{(\alpha)}$ has the same projection on $\mathbb{L}$, namely $P^{*}$. Hence (A.25) holds for every $Q\in\mathbb{B}^{(\alpha)}$. Thus we only need to prove (A.25) for every $Q\in\text{cl}(\mathbb{B}^{(\alpha)})\setminus\mathbb{B}^{(\alpha)}$. Let $Q\in\text{cl}(\mathbb{B}^{(\alpha)})\setminus\mathbb{B}^{(\alpha)}$. There exists $\{Q_{n}\}\subset\mathbb{B}^{(\alpha)}$ such that $Q_{n}\rightarrow Q$. Hence, for any $P\in\mathbb{L}$,

[TABLE]

Taking limit as $n\to\infty$, we have (A.25)101010For a fixed $P$, $Q\mapsto B_{\alpha}(P,Q)$ is continuous as a function from $\mathcal{P}$ to $[0,\infty]$.. This completes the proof. ∎

The following theorem tells us that a reverse $B_{\alpha}$-projection on a non-normalized $\alpha$-power-law family can be turned into a forward $B_{\alpha}$-projection on the associated linear family. We shall refer this as the projection theorem for the $B_{\alpha}$-divergence.

Theorem 7**.**

Let $\alpha<1$. Let $\mathbb{B}^{(\alpha)}$ be as in Definition 3 and let $X_{1},\dots,X_{n}$ be an i.i.d. sample drawn according to a particular member of $\mathbb{B}^{(\alpha)}$. Let $\widehat{P}$ be the empirical probability measure of $X_{1},\dots,X_{n}$ and let

[TABLE]

where $\bar{f_{i}}=\frac{1}{n}\sum_{j=1}^{n}f_{i}(X_{j}),i=1,\dots,k$. Let $P^{*}$ be the forward $B_{\alpha}$-projection of $Q$ on $\widehat{\mathbb{L}}$. Then the following hold.

1. (i)

If $P^{*}\in\mathbb{B}^{(\alpha)}$, then $P^{*}$ is the reverse $B_{\alpha}$-projection of $\widehat{P}$ on $\mathbb{B}^{(\alpha)}$. 2. (ii)

If $P^{*}\notin\mathbb{B}^{(\alpha)}$, then $\widehat{P}$ does not have a reverse $B_{\alpha}$-projection on $\mathbb{B}^{(\alpha)}$. However, $P^{*}$ is the reverse $B_{\alpha}$-projection of $\widehat{P}$ on cl$(\mathbb{B}^{(\alpha)})$.

Proof.

Let us first observe that $\widehat{\mathbb{L}}$ is so constructed that $\widehat{P}\in\widehat{\mathbb{L}}$. Since the families $\widehat{\mathbb{L}}$ and $\mathbb{B}^{(\alpha)}$ are defined by the same functions $f_{i}$, $i=1,\dots,k$, by Corollary 1, we have $\widehat{\mathbb{L}}\cap\text{cl}(\mathbb{B}^{(\alpha)})=\{P^{*}\}$ and

[TABLE]

Hence it is clear that the minimizer of $B_{\alpha}(\widehat{P},Q)$ over $Q\in\text{cl}(\mathbb{B}^{(\alpha)})$ is same as the minimizer of $B_{\alpha}(P^{*},Q)$ over $Q\in\text{cl}(\mathbb{B}^{(\alpha)})$ (Notice that this statement is also true with $\text{cl}(\mathbb{B}^{(\alpha)})$ replaced by $\mathbb{B}^{(\alpha)}$). But $B_{\alpha}(P^{*},Q)$ over $Q\in\text{cl}(\mathbb{B}^{(\alpha)})$ is uniquely minimized by $Q=P^{*}$. Hence, if $P^{*}\notin\mathbb{B}^{(\alpha)}$, since minimum of $B_{\alpha}(\widehat{P},Q)$ over $Q\in\text{cl}(\mathbb{B}^{(\alpha)})$ is same as the minimum of $B_{\alpha}(\widehat{P},Q)$ over $Q\in\mathbb{B}^{(\alpha)}$, the later is not attained on $\mathbb{B}^{(\alpha)}$. ∎

Remark 4**.**

Theorems 6, 7, and Corollary 1 continue to hold for $\alpha>1$ as well if attention is restricted to probability measures with strictly positive components and the existence of $P^{*}$ is guaranteed.

Appendix B The Map $\theta\mapsto\Phi(\theta)$ Defined in (2.24) is one-one:

Let us first recall that $\Phi=(\phi_{1},\dots,\phi_{k})$, where

[TABLE]

We will show that the determinant of the Jacobian matrix of $\Phi$ is non-zero. Then the claim follows from [1, Th. 13.6]. Let $\partial_{j}$ denote $\partial/\partial\theta_{j}$. Then the $(i,j)$th entry of the Jacobian matrix of $\Phi$ is given by

[TABLE]

We now calculate each of the terms in (B).

[TABLE]

Hence

[TABLE]

where we used the fact that

[TABLE]

Similarly

[TABLE]

Also

[TABLE]

where in (B) we used the fact that

[TABLE]

by (2.23) and (2.24).

Using (B.3) and (B), the last two terms in (B) together yield

[TABLE]

Substituting (B) and (B.5) in (B), we get

[TABLE]

Let $A$ denote the above covariance matrix, that is, the $(i,j)$th entry of $A$ is

[TABLE]

Then $A$ is symmetric and positive semi-definite. Let $B$ denote the Jacobian matrix of $\Phi$. Then, from (B.6), we have $B=-Z(\theta)^{1-\alpha}\overline{{P}_{\theta}^{\alpha-1}}A$. So, $B$ is also symmetric but negative semi-definite. If we can show that $B$ is negative definite then then the determinant of $B$ is non-zero. Thus it suffices to show that $A$ is positive definite. Suppose that $A$ is not positive definite. Then there exists a non-zero vector $c=(c_{1},\dots,c_{k})^{T}$ such that

[TABLE]

This implies

[TABLE]

Therefore,

[TABLE]

where $t=\mathbb{E}_{\theta^{(\alpha)}}[c_{1}P_{\theta}(X)^{1-\alpha}f_{1}(X)+\cdots+c_{k}P_{\theta}(X)^{1-\alpha}f_{k}(X)]$.

[TABLE]

As $P_{\theta}\in\mathbb{M}^{(\alpha)}$, we have

[TABLE]

for some $\theta=(\theta_{1},\dots,\theta_{k})\in\Theta$. Substituting this in (B.9), we get

[TABLE]

Observe that, with out loss of generality, we can assume that the vectors $f_{1}(\cdot),\dots,f_{k}(\cdot)$ in the $\alpha$-power-law family are linearly independent. Further, in view of footnote 6, we claim that the vector $Q(\cdot)^{\alpha-1}$ is linearly independent of the vectors $f_{1}(\cdot),\dots,f_{k}(\cdot)$. If not, let $Q(\cdot)^{\alpha-1}=c_{1}f_{1}(\cdot)+\cdots+c_{k}f_{k}(\cdot)$ for some non-zero vector $c=(c_{1},\dots,c_{k})^{T}$. Then, for any $P\in\mathbb{L}$, $\sum\limits_{x\in\mathcal{X}}P(x)Q(x)^{\alpha-1}=\sum\limits_{x\in\mathcal{X}}P(x)[c_{1}f_{1}(x)+\cdots+c_{k}f_{k}(x)]=0$. This is not possible since $Q$ has full support. Then, from (B), we have $t=0$ and hence $c_{1}=\cdots=c_{k}=0$, which is a contradiction. Hence $A$ is positive definite. This completes the proof.

Acknowledgements

Atin Gayen is supported by an INSPIRE fellowship of the Department of Science and Technology, Government of India.