Algebraic Properties of Wyner Common Information Solution under Graphical Constraints

TL;DR

This paper investigates the algebraic structure of the CMDFA solution for Gaussian vectors with a star graphical constraint, revealing conditions for rank-specific solutions and demonstrating differences in mutual information.

Contribution

It characterizes the solution space of CMDFA under star topology constraints, identifying conditions for rank one or rank n-1 solutions and analyzing their properties.

Findings

Solution is either rank one or rank n-1 under certain conditions.

Derived necessary and sufficient conditions based on edge weights.

Numerical results show differences between optimal and naive star solutions.

Abstract

The Constrained Minimum Determinant Factor Analysis (CMDFA) setting was motivated by Wyner's common information problem where we seek a latent representation of a given Gaussian vector distribution with the minimum mutual information under certain generative constraints. In this paper, we explore the algebraic structures of the solution space of the CMDFA, when the underlying covariance matrix $\Sigma_x$ has an additional latent graphical constraint, namely, a latent star topology. In particular, sufficient and necessary conditions in terms of the relationships between edge weights of the star graph have been found. Under such conditions and constraints, we have shown that the CMDFA problem has either a rank one solution or a rank $n-1$ solution where $n$ is the dimension of the observable vector. Numerical results are provided to demonstrate the difference between the optimal mutual information and that derived under a naive star constraint.

Equations

$$|{\alpha }_1|\ge |{\alpha }_2|\ge \cdots \ge |{\alpha }_n|$$

$${\Sigma }_x=\left[\begin{array}{cccc}1& {\alpha }_1{\alpha }_2& \dots & {\alpha }_1{\alpha }_n\\ {\alpha }_2{\alpha }_1& 1& \dots & {\alpha }_2{\alpha }_n\\ ⋮& ⋮& \ddots & ⋮\\ {\alpha }_n{\alpha }_1& {\alpha }_n{\alpha }_2& \dots & 1\end{array}\right]$$

$$\left[\begin{array}{c}{X}_1\\ ⋮\\ X_n\end{array}\right]=\left[\begin{array}{c}{\alpha }_1\\ ⋮\\ {\alpha }_n\end{array}\right]Y+\left[\begin{array}{c}{Z}_1\\ ⋮\\ Z_n\end{array}\right]$$

$$\Rightarrow$$

$$p(X_1{X}_2,\dots ,X_n|Y)={\Pi }_{i=1}^{n}p(X_i|Y)$$

$${\Sigma }_x=({\Sigma }_x-D)+D$$

$${\Sigma }_{t,ND}=\left[\begin{array}{cccc}{\alpha }_1^2& {\alpha }_1{\alpha }_2& \dots & {\alpha }_1{\alpha }_n\\ {\alpha }_2{\alpha }_1& {\alpha }_2^2& \dots & {\alpha }_2{\alpha }_n\\ ⋮& ⋮& \ddots & ⋮\\ {\alpha }_n{\alpha }_1& {\alpha }_n{\alpha }_2& \dots & {\alpha }_n^2\end{array}\right]$$

$${\theta }_1\ge {\theta }_2\ge \cdots \ge {\theta }_n$$

$${\theta }_1\le \sum _{i=2}^n{\theta }_i$$

$${\Sigma }_{t,ND}T=\vec{0}$$

$$\Rightarrow$$

$$||{\alpha }_i{\vec{t}}_{i,\ast }|{|}^2={\alpha }_i^2||{\vec{t}}_{i,\ast }|{|}^2||{\alpha }_i^2\sum _{j=1}^n{t}_{ij}^2=\frac{{\alpha }_i^2}{1-{\alpha }_i^2},1\le i\le n$$

$$|{\alpha }_1|||{\vec{t}}_{1,\ast }||\le \sum _{i=2}^n|{\alpha }_i|||{\vec{t}}_{i,\ast }||$$

$$\Rightarrow$$

$$v_1=\left[\begin{array}{c}-\frac{{\alpha }_2}{{\alpha }_1}\\ 1\\ 0\\ ⋮\\ 0\end{array}\right],v_2=\left[\begin{array}{c}-\frac{{\alpha }_3}{{\alpha }_1}\\ 0\\ 1\\ ⋮\\ 0\end{array}\right],\dots ,v_{n-1}=\left[\begin{array}{c}-\frac{{\alpha }_n}{{\alpha }_1}\\ 0\\ 0\\ ⋮\\ 1\end{array}\right]$$

$$V$$

$$=\left[\begin{array}{cccc}-\frac{{\alpha }_2}{{\alpha }_1}& \dots & -\frac{{\alpha }_n}{{\alpha }_1}& -\left(c_2\frac{{\alpha }_2}{{\alpha }_1}+\cdots +c_n\frac{{\alpha }_n}{{\alpha }_1}\right)\\ 1& \dots & 0& c_2\\ 0& \dots & 0& c_3\\ ⋮& \ddots & ⋮& ⋮\\ 0& \dots & 1& c_n\end{array}\right]$$

$$T_{n\times n}=V_{n\times n}\cdot B_{n\times n}$$

$$T{T}^{\prime }=VB{B}^{\prime }{V}^{\prime }=V\beta V^{\prime }$$

$$\frac{{\alpha }_2^2}{{\alpha }_1^2}{\beta }_{11}+\frac{{\alpha }_3^2}{{\alpha }_1^2}{\beta }_{22}+\cdots +\frac{{\alpha }_n^2}{{\alpha }_1^2}{\beta }_{n-1,n-1}+$$

$${\left(c_2\frac{{\alpha }_2}{{\alpha }_1}+c_3\frac{{\alpha }_3}{{\alpha }_1}+\cdots +c_n\frac{{\alpha }_n}{{\alpha }_1}\right)}^2{\beta }_{nn}=\frac{1}{1-{\alpha }_1^2}$$

$${\beta }_{ii}+c_{i+1}^2{\beta }_{nn}=\frac{1}{1-{\alpha }_{i+1}^2},i=1,\dots ,n-1$$

$${\beta }_{nn}=$$

$$c_i{\alpha }_i=\frac{{\tilde{c}}_i{\alpha }_i}{\sqrt{1-{\alpha }_i^2}}={\theta }_i,i=2,\dots ,n$$

$${\beta }_{nn}=\frac{{\theta }_1^2-{\theta }_2^2-\cdots -{\theta }_n^2}{{\sum }_{i\ne j,i\ne 1,j\ne 1}{\theta }_i{\theta }_j}$$

$${\theta }_1^2\le {\left(\sum _{i=2}^n{\theta }_i\right)}^2$$

$$\Rightarrow$$

$$\sum _{i=1}^n{\alpha }_i{\vec{t}}_{i,\ast }=\vec{0}$$

$$||{\alpha }_1{\vec{t}}_{1,\ast }||=||\sum _{i=2}^n{\alpha }_i{\vec{t}}_{i,\ast }||$$

$${\alpha }_1{\vec{t}}_{1,\ast }=-\sum _{i=2}^n{\alpha }_i{\vec{t}}_{i,\ast }$$

$$||\sum _{i=2}^n{\alpha }_i{\vec{t}}_{i,\ast }||\le \sum _{i=2}^n||{\alpha }_i{\vec{t}}_{i,\ast }||$$

Full text

11footnotetext: Md M Hasan, S. Wei and A. Moharrer are with the school of Electrical Engineering and Computer Science, Louisiana State University, Baton Rouge, LA 70803, USA (Email: mhasa15@lsu.edu, swei@lsu.edu, alimoharrer@gmail.com).

I INTRODUCTION

Factor Analysis (FA) is a commonly used tool in multivariate statistics to represent the correlation structure of a set of observables in terms of significantly smaller number of variables called “latent factors”. With the growing use in data mining, high dimensional data analytics, factor analysis has already become a prolific area of research [1][2]. Classical Factor Analysis models seek to decompose the correlation matrix of an $n$-dimensional random vector ${\bf X}\in{\mathcal{R}}^{n}$, $\Sigma_{x}$, as the sum of a diagonal matrix $D$ and a Gramian matrix $\Sigma_{x}-D$.

The literature that approached Factor Analysis can be classified in three major categories. Firstly, algebraic approaches [3] and [4], where the principal aim was to give a characterization of the vanishing ideal of the set of symmetric $n\times n$ matrices that decompose as the sum of a diagonal matrix and a low rank matrix, did not offer scalable algorithms for higher dimensional statistics. Secondly, Factor Analysis via heuristic local optimization techniques, often based on the expectation maximization algorithm, were computationally tractable but offered no provable performance guarantees. The third and final type are the convex optimization based methods such as Constrained Minimum Trace Factor Analysis (CMTFA) [5] [6] and CMDFA [7]. The motivation behind CMDFA comes from Wyner’s common information $C(X_{1},X_{2})$ which characterizes the minimum amount of common randomness needed to approximate the joint density between a pair of random variables $X_{1}$ and $X_{2}$ to be $C(X_{1},X_{2})=\min_{\underset{X_{1}-Y-X_{2}}{P_{Y}}}I(X_{1},X_{2};Y)$, where $I(X_{1},X_{2};Y)$ is the mutual information between $X_{1}$, $X_{2}$ and $Y$, $X_{1}-Y-X_{2}$ indicates the conditional independence between $X_{1}$ and $X_{2}$ given $Y$, and the joint density function is sought to esnure such conditional independence as well as the given joint density of $X_{1}$ and $X_{2}$. Since the Factor Analysis of the Gaussian random vector $\vec{X}$ can be modelled as $\vec{X}=A\vec{Y}+\vec{Z}$, where $A_{n\times k}$ is a real matrix, $\vec{Y}_{k\times 1},k<n$ is the vector of independent latent variables and $\vec{Z}_{n\times 1}$ is a Gaussian vector of zero mean and covariance matrix $\Sigma_{z}=D$. Hence we have, $I(\vec{X};\vec{Y})=h(\vec{X})-h(\vec{X}|\vec{Y})=h(\vec{X})-h(\vec{Z})$ where $I(\vec{X};\vec{Y})$ is the mutual information between $\vec{X}$ and $\vec{Y}$, $h(\vec{X}),h(\vec{Z})$ are differential entropies of $\vec{X}$ and $\vec{Z}$ and $h(\vec{X}|\vec{Y})$ is the differential entropy of $\vec{X}$ given $\vec{Y}$. Hence characterizing the common information between $\vec{X}$ and $\vec{Y}$ [8][9][10] would be $\min_{A,\Sigma_{z}}I(\vec{X};\vec{Y})$ which is an equivalent problem to $\max_{\Sigma_{z}}h(\vec{Z})$ hence equivalent to $\min_{\Sigma_{z}}-\log|\Sigma_{z}|$.

The scope of this paper is limited to analysing the solution space of CMDFA and recovering the underlying graphical structures. It is important to remark that our work is not concerned about the algorithm side of the optimization technique, rather our focus is to characterize and find insights about their solution space. Moharrer and Wei [7] derived CMDFA from a broader class of convex optimization problem and established a relationship between the outcome of these optimization techniques and common information problem [9]. We find the explicit conditions under which the CMDFA solution of $\Sigma_{x}$ recoves a star structure. Since star may not always be the optimum solution, we have also shown the existence and uniqueness of a rank $n-1$ CMDFA solution of $\Sigma_{x}$ which is the only other possible solution. We have shown analyticaly the optimality of the non-star solution over star under certain circumstances from common information point of view, and at the end presened some numerical data to support our claims.

II Definitions and Notations

Let $\vec{b}$ be a real $n$ dimensional column vector and $A$ be an $n\times n$ matrix. As in literature in general we denote the $i$th element $\vec{b}$ as $b_{i}$ and the $(i,j)$th element of $A$ as $A_{i,j}$. Here we define all the vector operations and notations in terms of $\vec{b}$ and $A$, that will carry their meaning on other vectors and matrices throughout this paper unless stated otherwise.

Let $\vec{a}_{i,*}$ and $\vec{a}_{*,i}$ denote the $i$th row and $i$th column vector of matrix $A$ respectively. Function $\lambda_{min}(A)$ is defined to be the smallest eigen value of matrix $A$. $N(A)$ stands for the null space of matrix $A$.

Vectors $\vec{1}$ and $\vec{0}$ are the $n$ dimensional column vectors with each element equal to $1$ and [math] respectively. When we write $\vec{b}\geq 0$ we mean that each element of the vector $b(i)\geq 0,1\leq i\leq n$. $\vec{b}^{2}$ is the Hadamard product of vector $\vec{b}$ with itself. $||\vec{b}||$ denotes the $L_{2}$ norm of vector $\vec{b}$.

Now we define two terms i.e. dominance and non-dominance of a vector which will repeatedly appear throughout the paper. When we talk about the dominance or non-dominance of any vector $\vec{b}$ we assume that the elements of the vector are sorted in a way such that $|b_{1}|\geq|b_{2}|\geq\dots\geq|b_{n}|$. We call vector $\vec{b}$ dominant and $b_{1}$ the dominant element if for the above sorted vector $|b_{1}|>\sum_{j\neq 1}|b_{j}|$ holds. Otherwise $\vec{b}$ is non-dominant.

III Formulation of the Problem

First of all we define the real column vector $\vec{\alpha}$ as $\vec{\alpha}=[\alpha_{1},\dots,\alpha_{n}]^{\prime}\in{\mathcal{R}}^{n}$ where $0<|\alpha_{j}|<1$, $j=1,2,\dots,n$ and

[TABLE]

Let us consider a star structured population covariance matrix $\Sigma_{x}$ having all the diagonal comptonents $1$ as given by equation (2).

[TABLE]

The above covariance matrix could be produced by the graphical model given by equation (3).

[TABLE]

where

• •

$\{X_{1},...,X_{n}\}$ are conditionally independent Gaussian random variables given $Y$, forming the jointly Gaussian random vector $\vec{X}\sim\mathcal{N}(0,\Sigma_{x})$ where $Y\sim\mathcal{N}(0,1)$.

• •

$\{Z_{1},...,Z_{n}\}$ are independent Gausian random varables with $Z_{j}\sim\mathcal{N}(0,1-\alpha_{j}^{2})\quad 1\leq j\leq n$ forming the Gaussian random vector $\vec{Z}$.

The above graphical model assumes the conditional independence among the observables given the latent variable given by (5) giving rise to a star topology.

[TABLE]

CMDFA aims to minimize the mutual information between the observable Gaussian random vector $\vec{X}$ and the latent ones $\vec{Y}$. It is thus to seek joint distribution between the latent and observable ones such that the differential entropy $h(X|Y)$ is maximized. Under the joint Gaussian distribution, it is the same as seeking factorization of $\Sigma_{x}$ such that the determinant of the matrix $D$ is minimized as in equation (6) under the constraint that both $(\Sigma_{x}-D)$ and $D$ are Gramian matrices.

[TABLE]

$\Sigma_{x}$ being produced by the model in (3) would equivalently mean (6) having a rank $1$ solution i.e. $\Sigma_{x}-D$ being $\Sigma_{t,ND}$ given by equation (7).

[TABLE]

But CMDFA solution of $\Sigma_{x}$ may not always be rank $1$, indicating star may not always be the optimum solution from common information point of view. It remains to be seen if CMDFA solution to $\Sigma_{x}$ recovers the graphical model given by (3). Also to be investigated is the exact solution to CMDFA if it fails to recover the underlying star topology. In the rest of the paper, we will present both sufficient and necessary conditions under which the rank of the optimal $\Sigma_{x}-D$ and the values of $D$’s entries are determined.

IV Solutions to CMDFA

In this section we present the detailed analysis of the CMDFA solution space of $\Sigma_{x}$. We defne the real column vector $\vec{\theta}\in{\mathcal{R}}^{n}$ as $\vec{\theta}=[\theta_{1},\dots,\theta_{n}]^{\prime}$ where $\theta_{i}=\frac{|\alpha_{i}|}{\sqrt{1-\alpha_{i}^{2}}},1\leq i\leq n$.

As we can see, each elements in $\vec{\theta}$ is equal to the square root of the signal to noise ratio ($\sqrt{\text{SNR}}$) of the corresponding element of vector $\vec{\alpha}$. The following order of the elements of $\vec{\theta}$ is a necessary consequence of our assumption in (1),

[TABLE]

As we metioned before, we are interested to find out if CMDFA low rank decomposition of $\Sigma_{x}$ produces a rank $1$ matrix. Next we analyse the solution space of CMDFA and find explicit conditions for both when the solution is rank $1$ and when it is not. To start the proceedings we state Theorem 1 given in [7] that gives the necessary and sufficient condition for $D^{*}$ to be the CMDFA solution of the decomposition given in (6).

Theorem 1**.**

The matrix $D^{*}$ is the CMDFA solution of $\Sigma_{x}$ if and only if $\lambda_{min}(\Sigma_{x}-D^{*})=0$, and there exists $n\times k$ matrix $T$ such that $\vec{t}_{*,i}\in\mathcal{N}(\Sigma_{x}-D^{*}),1\leq i\leq k$ and $||\vec{t}_{i,*}||^{2}=\frac{1}{1-\alpha_{i}^{2}},1\leq i\leq n$.

In the first of the two subsections of this section, we find the conditions under which CMDFA solution of $\Sigma_{x}$ recovers the model given by (3) or equivalently speaking, find condtions under which CMDFA solution of $\Sigma_{x}$ is the rank $1$ matrix given by (7). In the other subsection, we show the detailed analysis on the existance and uniqueness of the CMDFA solution of $\Sigma_{x}$, when the solution is not a rank $1$ matrix.

IV-A CMDFA Non-dominant Case

Here we analyse the conditions under which the CMDFA solution of $\Sigma_{x}$ recovers a star structure. Lemma 1 sets the groundwork for the Theorem to follow. The Lemma also has a geometric interpretaion that enriches our overall understanding of the CMDFA non-dominant case.

Lemma 1**.**

There exists $n\times r$ matrix $T$ such that $\vec{t}_{*,i}\in N(\Sigma_{t,ND}),\quad 1\leq i\leq r$ and $||\vec{t}_{j,*}||^{2}=\frac{1}{1-\alpha_{j}^{2}},\quad 1\leq j\leq n$ if and only if vector $\vec{\theta}$ is non-dominant i.e.,

[TABLE]

Proof of Lemma 1:.

Let $\vec{t}_{i,*},1\leq i\leq n$ be the $i$th row vector of the matrix $T$ and $\vec{0}$ denote the zero column vector. We need,

[TABLE]

Equation (10) has a beautiful geometric interpretation. The length of each vector $\alpha_{i}\vec{t}_{i,*}$ can be written as,

[TABLE]

Hence, each vector $\alpha_{i}\vec{t}_{i,*}$ is a point on the surface of an $n$ dimensional sphere of radius $\frac{|\alpha_{i}|}{\sqrt{1-\alpha_{i}^{2}}}$. Equation (10) dictates that, for the matrix $T$ to be in the null space of $\Sigma_{t,ND}$ the biggest of those spheres can not have a radius greater than the sum of the other radiuses. This leads us to the condition of non-dominance. Using (10),

[TABLE]

∎

For further clarification, we refer to the co-centric spheres (assuming $n=3$) in Figure 1. Let $||\vec{OA}||=\theta_{1}$, $||\vec{OP}||=\theta_{2}$, $||\vec{OQ}||=\theta_{3}$. If $\theta_{1}>\theta_{2}+\theta_{3}$, it is impossible to find any vector on the outer most sphere that can be expressed as the vector sum of vectors $\vec{OP}$ and $\vec{OQ}$. On the other hand if $\theta_{1}\leq\theta_{2}+\theta_{3}$ proper selection of angles $\sigma_{2}$ and $\sigma_{3}$ can always ensure $\vec{OA}$ be a vector sum of $\vec{OP}$ and $\vec{OQ}$ or equivalently ensure the orthogonality given by $\vec{OB}+\vec{OP}+\vec{OQ}=\vec{0}$.

Having proved Lemma 1 we are now well equipped to state and prove the statement of Theorem 2 that has the main result of this subsection.

Theorem 2**.**

CMDFA solution of $\Sigma_{x}$ is $\Sigma_{t,ND}$ if and only if $\vec{\theta}$ is non-dominant.

The theorem states that the CMDFA solution to a star connected network is a star itself, if and only if there is no dominant element in the vector $\vec{\theta}$.

Proof of Theorem 2:.

Now we refer back to the necessary and sufficient condition for CMDFA solution at the begining of this section given by Theorem 1. Since, $\Sigma_{t,ND}$ in rank $1$, its minimum eigenvalue is [math]. To complete the proof of Theorem 2, we only need to show the existance of matrix $T$ such that the column vectors of $T$ are in the null space of $\Sigma_{t,ND}$ and the $L_{2}$-norm square of the $i$th row of $T$ is $\frac{1}{1-\alpha_{i}^{2}},1\leq i\leq n$.

Lemma 1 has already shown that, for the existence of such $T$ non-dominance given by equation (9) is a necessary condition. Next we show, by constructing such a $T$ matrix under the assumption of non-dominance of $\vec{\theta}$, that non-dominace is also a sufficient condition . And that should complete the proof of Theorem 2.

It is straightforward to find the following basis vectors for the null space of $\Sigma_{t,ND}$,

[TABLE]

We define matrix $V$ so that its columns span the null space of $\Sigma_{t,ND}$,

[TABLE]

where $c_{i}=\frac{\widetilde{c}_{i}}{\sqrt{1-\alpha_{i}^{2}}},\quad i=2,\dots,n$ and $\widetilde{c}_{i}\in\{1,-1\}$.

The columns of $V$ span the null space of $\Sigma_{t,ND}$. To construct our desired matrix $T$, under the assumption of non-dominance of $\vec{\theta}$, it will suffice for us to find a diagonal matrix $B_{n\times n}$ such that the following holds.

[TABLE]

where the $L_{2}$-norm square of the $i$th row of $T$ is $\frac{1}{1-\alpha_{i}^{2}}$. Using (14),

[TABLE]

We require the diagonal matrix $\beta$ to have only non-negative entries. Based on the conditions imposed on the matrix $T$, we have the following $n$ equations,

[TABLE]

[TABLE]

Solving, (IV-A) with the help of (17) we get,

[TABLE]

It is straightforward to see that, to ensure all the $\beta_{\textbullet ii},1\leq i\leq n$ are non-negative, we need $\beta_{nn}\leq 1$. We select $\widetilde{c}_{i},2\leq i\leq n$ such that,

[TABLE]

Under such selection of $\widetilde{c}_{i},2\leq i\leq n$, $\beta_{nn}$ becomes,

[TABLE]

Now, using the non-dominance assumption given in (9), we have

[TABLE]

Which means non-dominance of vector $\vec{\theta}$ is a sufficient condition to construct the kind of $T$ matrix we are looking for. That completes the proof of Theorem 2. ∎

Boundary Case

It is obvious that, there might be numerous ways to construct the matrix $T$ that satisfisfy the requirements set by Theorem 1. Because of the special way we constructed the matrix $T$ the rank of $T$ under the non-dominant case is $n-1$ except for a very special boundary case. Under the boundary case i.e. when the inequality (9) holds for equality, when the rank of $T$ is always $1$ irrespective of the way we construct $T$ . For any given $n$, it is straightforward to see from equation (21) that, for $\theta_{1}=\sum_{i=2}^{n}\theta_{i}$ we have $\beta_{nn}=1$. Plugging $\beta_{nn}=1$ in equation (17) gives us $\beta_{ii}=0,2\leq i\leq n$. Equations (14) and (15) suggest that, such a $\beta$ matrix will produce a rank $1$ matrix $T$. This very special case is explained by the next Lemma.

Lemma 2**.**

When the non-dominance condition given in (9) holds for equality, any $n\times r$ matrix $T$ such that $\vec{t}_{*,i}\in N(\Sigma_{t,ND}),\quad 1\leq i\leq r$ and $||\vec{t}_{j,*}||^{2}=1,\quad 1\leq j\leq n$ has to be a rank $1$ matrix.

Proof of Lemma 2:.

Using the orthogonality between $\Sigma_{t,ND}$ and its null space matrix $T$,

[TABLE]

Equation (23) implies the following two things:

[TABLE]

Using the triangular inequality,

[TABLE]

If all the $\alpha_{i}\vec{t}_{i,*},2\leq i\leq n$ are not in the same direction, the the above inequality becomes

[TABLE]

Hence, under the boundary condition i.e. $\theta_{1}=\sum_{i=2}^{n}\theta_{i}$ , we have

[TABLE]

Which violets (24). That means to ensure $||\alpha_{1}\vec{t}_{1,*}||=||\sum_{i=2}^{n}\alpha_{i}\vec{t}_{i,*}||$, all of $\alpha_{i}\vec{t}_{i,*},2\leq i\leq n$ have to be in the same direction. This along the second implication of orthogonality given by equation (25), makes matrix $T$ a rank $1$ matrix. ∎

IV-B Dominant Case

Having proved that the non-dominance of vector $\vec{\theta}$ is a sufficient and necessary condition for CMDFA solution of $\Sigma_{x}$ to recover a star structure, we are left with only the dominance case now i.e.

[TABLE]

Under the above dominant condition we want to show the existence of a rank $n-1$ solution of $\Sigma_{x}$. Any solution we find will be unique, because CMDFA is a special type of the broader class of convex optimization problem defined in [11]. We still have to satisfy the same sufficient and necessary condtion for the CMDFA solution, that we presented at the begining of this section. Like the non-dominant case, for the matrix $D^{*}$ to be the CMDFA solution of $\Sigma_{x}$ under the dominant case, the minimum eigen value of $\Sigma_{x}-D^{*}$ has to be $\lambda_{min}(D^{*})=0$ and the $L_{2}$-norm square of the $i$th row of the null space matrix $T$ has to be $\frac{1}{1-\alpha_{1}^{2}}$. The only difference with the non-dominant case is that, since our conjecture for the dominant case is an $n-1$ rank solution, the null space matrix $T$ will always be rank $1$ i.e. a column vector. Mathematically speaking, we need to show the existance of $0<a_{i}<1,1\leq i\leq n$ such that the following orthogonality condition holds.

[TABLE]

where $c_{i}\in\{-1,1\}$. Once we have such $a_{i},1\leq i\leq n$ the $i$th element of the CMDFA solution vector $\vec{d^{*}}$ under the dominant case will be $1-a_{i},1\leq i\leq n$. The above orthogonality relationship gives us the following $n$ equations.

[TABLE]

Let $(i)$ denote the $i$th equation given by (30). Using the linear combination $\alpha_{i+1}\times(i)-\alpha_{i}\times(i+1),1\leq i\leq n$ gives us the following $n-1$ equations.

[TABLE]

where

[TABLE]

Equation (51) implies that for some ratio $\mu$ we can write the following,

[TABLE]

Now plugging the expressions from (32) and (33) in any of the $n$ equations given by (30) we get,

[TABLE]

It will suffice for us to prove the existence of $0<a_{i}<1,\quad 1\leq i\leq n$ such that (34) holds. From the definition of $\eta_{i}$ given in (32) we see that, to find each $a_{i},\quad 1\leq i\leq n$ we need to solve the following second order polynomial.

[TABLE]

If we solve equation (35) for each $a_{i}$ we will get a left root and a right root. Our initial conjecture is that the left root for $a_{1}$ and right roots for $a_{2},\dots,a_{n}$ that we get solving (35) will give us $0<a_{i}<1,\quad 1\leq i\leq n$ that satisfy (34). If we can prove that our conjecture is true, then that should be the only possible solution to (34) because of the uniqueness of solution to such convex optimization problems proved in [11]. Plugging in the left root for $a_{1}$, right roots for $a_{2},\dots,a_{n}$ in (34) gives us the following equation.

[TABLE]

We define

[TABLE]

Under these newly defined $X_{i}$s (IV-B) becomes,

[TABLE]

And using the definition of $X_{i},1\leq i\leq n$ given in (37), we get the following cylinders of hyperbolas.

[TABLE]

Equations given by (39) imply that for each value of $X_{1}$ we get a point $[X_{1},X_{2},\dots,X_{n}],$ in the $n$ dimensional space where each $X_{i},2\leq i\leq n$ is a function of $X_{1}$. For the range of values of $(\frac{1}{2}<X_{1}<\infty)$ all such points together produce an $n$ dimensional space curve. If we project this space curve on any of the two dimensional $(X_{1},X_{i}),2\leq i\leq n$ planes we get a hyperbola.

Another important thing to note is that, each equation given by (39) is a cylinder of hyperbolas originated from $(X_{1},X_{i})$ plane and projected onto $n$ dimensional space. Each point in the space curve represents an intersection points of all $n-1$ cylinders of hyperbolas originated from $(X_{1},X_{i}),2\leq i\leq n$ planes.

At this point our revised goal is to show the existence of a point in the space curve that satisfies equation (38) under the dominance condition given by (28). Becasue of the way we defined $X_{i}$s $1\leq i\leq n$ the solution must satisfy the condition $X_{i}>\frac{1}{2},1\leq i\leq n$. Theorem 3 states the main result of this subsection.

Theorem 3**.**

There exists an intersection point among the plane given by (38) and the $n-1$ cylinders of hypberbolas given by (39), that satisfies $X_{i}>\frac{1}{2},1\leq i\leq n$.

Proving the above Theorem would mean that, there exists $0<a_{i}<1,\quad 1\leq i\leq n$ such that (34) holds, which in turn would mean the existance of an $n-1$ rank CMDFA solution under the dominance of vector $\vec{\theta}$. And as we mentioned already, the uniqueness of such solution is guaranteed.

Proof of Theorem 3:.

Let us define the function $G(.)$ of $X_{1}$ as the inner product between the vectors $[X_{1},\dots,X_{n}]$ and $[\theta_{1}^{2},\dots,\theta_{n}^{2}]^{\prime}$ where each $X_{i},1\leq i\leq n$ is a function of $X_{1}$. Which means,

[TABLE]

So, our revised goal becomes to find the existence of such $X_{1}>\frac{1}{2}$ for which the function of $G(X_{1})$ becomes $G(X_{1})=1+\frac{1}{2}\sum_{i=1}^{n}\theta_{i}^{2}$. And to achieve that goal some functional analysis of $G(X_{1})$ that we present next are of paramount importance.

Equation (39) dictates that each $X_{i}(X_{1}),2\leq i\leq n$ is a concave function of $X_{1}>\frac{1}{2}$. Which makes $G(X_{1})$ given by (40) a convex function of $X_{1}$ as the sum of convex functions of $X_{1}$. Using (39) and (40) we get,

[TABLE]

Using (37) we get,

[TABLE]

where, $\nu=\frac{1}{\lambda^{2}}$. Using (40) and (42),

[TABLE]

Hence,

[TABLE]

which is a negative value. We define $\hat{X}_{1}$ such that,

[TABLE]

Figures 2 and 3 illustrate our findings from the above functional analysis. As each $X_{i}(X_{1}),2\leq i\leq n$ is an increasing function of $X_{1}$ the ratios $\frac{X_{1}}{X_{i}(X_{1})},2\leq i\leq n$ are decreasing functions of $X_{1}$. Hence equation (43) suggests that $\frac{dG(X_{1})}{dX_{1}}$ is an increasing function of $X_{1}$. Given that knowledge, equations (44) and (45) considered together imply $\hat{X}_{1}>\frac{1}{2}$ as seen in Figures 2 and 3. One important to remark is that we see the function $\frac{dG(X_{1})}{dX_{1}}$ gets saturated gradually and is upperbounded by a value. This is because the ratio $\frac{X_{i}(X_{1})}{X_{1}},2\leq i\leq n$ is the slope of hyperbola in $X_{1}-X_{i}$ plane which is upper bounded by $\frac{\theta_{1}}{\theta_{i}}$ which is the slope of the asyptote in the respective plane. Plugging these individual upperbounds in (42) we get the dotted upper bound in Figure 3.

We can argue, as we refer to Figure 2, since $G$ is a convex function of $X_{1}$, it must be an increasing function for the values $X_{1}>\hat{X}_{1}$. Equations (45) and (41) imply that the convex function value $G(\hat{X}_{1})<G\left(\frac{1}{2}\right)<1+\frac{1}{2}\sum_{i=1}^{n}\theta_{i}^{2}$. Hence, there must exist $X_{1}^{*}>\hat{X}_{1}>\frac{1}{2}$ such that $G(X_{1}^{*})=1+\frac{1}{2}\sum_{i=1}^{n}\theta_{i}^{2}$. ∎

From Theorem 3 and its proof we know that the solution $X_{1}^{*}$ produces a corresponding $n$ dimensional point $\vec{X}^{*}=[X_{1}^{*},\dots,X_{n}^{*}]^{\prime}$ in the space curve which is the intersection point among hyperbolic cylinders and the plane given by (39) and (38) respectively. If we reflect on the bigger picture, $X_{1}^{*}$ plugged in (37) will produce a value of $\mu$ which in turn plugged in (35) will give us a set of $a_{i},1\leq i\leq n$ that satisfies (29).

IV-B1 Bounds of the Solution

Here we find a lowerbound and an upperbound to $X_{1}^{*}$.

Upperbound to $X_{1}^{*}$: It is easy to derive that the $i$th hyperbolic cylinder given by (39) has the following corresponding equation of the cylinder asymptotes (the ones passing through the origin and the first quadrant of the respective plane).

[TABLE]

Solving (46) and (38) together we get a value of $X_{1}$ which we denote as $X_{1}^{up}$ given by (47),

[TABLE]

Substituting $X_{1}$ in (46) by $X_{1}^{up}$ gives us a vector $\vec{X}^{up}=[X_{1}^{up},\dots,X_{n}^{up}]^{\prime}$ in the $n$ dimensional space, which is the intersection of the cylinders of asymptotes in (46) and the plane in (38).

Lemma 3**.**

The intersection point among the plane in (38) and the hyperbolic cylinders in (39) is upperbounded by the intersection point among the same plane and asymptotes of the respective hyperbolic cylinders given by (46),

The prooof of Lemma 3 is given in Appendix A. According the statement of this Lemma $\vec{X}^{up}>\vec{X}^{*}$. Which immediately suggests that $X_{1}^{up}$ given by (47) is an upperbound on $X_{1}^{*}$.

Lowerbound to $X_{1}^{*}$: We see in Figure 2 that the average slope of the curve $ABC$ is captured by the slope of the line $AC$. We assume that the dashed line $AD$ in Figure 2 has slope $\theta_{1}\left(\theta_{1}-\sum_{i=2}^{n}\theta_{i}\right)$ i.e. the upperbound of $\frac{dGX_{1}}{dX_{1}}$ given in Figure 3. Figure 3 suggests that, the slope at each point of the curve $ABC$ is strictly less than the slope of $AD$ in Figure 2, hence the slope of $AC$ must be less than the slope of $AD$. Now considering triangles $\triangle ADE$ and $\triangle ACF$ in Figure 2 we have,

[TABLE]

Which suggests $X_{1}^{low}$ a lowerbound of the actual $X_{1}^{*}$. Next we find the expression for $X_{1}^{low}$ using the geometry in Figure 2.

[TABLE]

V Numerical Data

We motivated CMDFA in terms common information which is a function of the minimum mutual information between the observables and the latent factors. It is a common practice to assume the star topology i.e the assumption that all the observables are mutually independent given a latent factor. Though star offers a sparce structure and smooth analysis, it may not be always the optimum solution. Next we show that assumption of star under CMDFA dominant case does not produce optimum outcome from common information point of view. We show that under the dominant case CMDFA solution provides lower mutual information between the observables and the latent variables that the star solution. Which in turn means lower common randomness required to produce the joint distribution between the observables and the latent variables and hence lower Wyner common information. In summary, we are about to demonstrate the additional cost in using more information bits to synthesize $n$-dimensional Gaussian vector under a star topology, when we do not use the solution of CMDFA, under the dominant case.

As mentioned before, each of $X_{1}^{low}$ and $X_{1}^{up}$ will produce a corresponding $\mu$ from equaion (37) and a set $a_{i},1\leq i\leq n$ or equivalently produce a matrix $\Sigma_{z}$ that decomposes (6). Let $X_{1}^{low}$ and $X_{1}^{up}$ produce $\mu^{low}$ and $\mu^{up}$ from equaion (37), the corresponding sets $\{a_{i}^{low}\}_{i=1}^{n}$ and $\{a_{i}^{up}\}_{i=1}^{n}$ from (35), corresponding matrices $\Sigma_{z}^{low}$ and $\Sigma_{z}^{up}$ that decompose (6), and $I^{low}$, $I^{up}$ be the corresponding mutual information between observed variables and the latent variables respectively. Also let $\Sigma_{z}^{star}$ be the solution to (6) when the CMDFA solution is a star and $I^{star}$ be the corresponding mutual information between the observed variables and the latent factor . Next Theorem analytically shows that each of $I^{low}$, $I^{up}$ produces better results than $I^{star}$ considered from common information point of view. We present the comparative results with respect to the varying magnitude of the dominance of $\vec{\theta}$. Referring to equation (28), we vary the dominance of vector $\vec{\theta}$ by changing the value of the first element $\theta_{1}$ while keeping other elements unchanged.

Theorem 4**.**

All of $I^{star}-I^{low}$, $I^{star}-I^{up}$ and $I^{up}-I^{low}$ are increasing functions of $\theta_{1}$

The proof of Theorem 4 is given in Appendix B. Figure 4 presents $I^{star}-I^{low}$, $I^{star}-I^{up}$ and $I^{star}-I^{CMDFA}$ as functions of $\theta_{1}$ for a particular case of $n=3$, where $I^{CMDFA}$ is the mutual information between the observed variables and the latent ones corresponding to the numerically found solution $X_{1}^{*}$. As we mentioned in the introduction that the primary motivation for this part of the work comes from common information, and the fact that in general people tend to assume a star topology to find common information, any value of mutual information less than $I^{star}$ works to our advantage. $I^{star}-I^{up}$ is an increasing function of $\theta_{1}$ indicates that the lower bound of the advantage of CMDFA solution over star increases as vector $\vec{\theta}$ becomes more and more dominant. We numerically calculated $I^{CMDFA}$ and the curve in Figure 4 gives the actual advantage that CMDFA soution has over star under the dominance of $\vec{\theta}$ whereas $I^{star}-I^{low}$ gives an upperbound to the actual advantage of CMDFA over a star topology. The gap between $I^{star}-I^{low}$ and $I^{star}-I^{up}$ is gradually increasing indicating $I^{up}-I^{low}$ is increasing with $\theta_{1}$ which justifies the statement of Theorem 4.

VI Conclusion

In this paper we analyzed the solution spaces of convex optimization algorithm CMDFA. We found conditions under which the solution is a star (rank $1$) and proved the existence of a rank $n-1$ solution when the solution is not a star. Through analytical analysis followed by numerical data we showed that star is not always the optimum solution. We particularly demonstrated the additional cost in using more information bits to synthesize $n$-dim Gaussian vector under a star topology, when we do not use the solution of CMDFA, under the dominant case.

Appendix A •

Proof of Lemma 3:.

Let $\vec{X}^{*}=[X_{1}^{*},\dots,X_{n}^{*}]^{\prime}$ be the CMDFA solution vector i.e. the intersection point of (39) and (38). We refer to Figure 5, the CMDFA dominant case solution vector $\vec{X}^{*}=[X_{1}^{*},\dots,X_{n}^{*}]^{\prime}$ has been projected on the $(X_{1},X_{i})$ plane which is shifted in the direction of $X_{j},j\neq 1,j\neq i$ by $X_{j}^{*}$. The procection of the $n$ dimensional plane given by (38) on this $(X_{1},X_{i})$ plane is given by the line $DPQ$ which is at a perpendicular distance $OD$ (because $\angle ODP=90^{\circ}$) from the origin. Here $OD$ is the projection of the vector $[\theta_{1}^{2},-\theta_{2}^{2},-\dots,-\theta_{n}^{2}]^{\prime}$ on $(X_{1},X_{i})$ plane, whose length we can calculate from equation (38) as,

[TABLE]

Geometrically, we can see in Figure 5 that the line $OPQ$ which is the projection of the plane cuts the hyperbola at point $P$ and the corresponding asymptote at point $Q$ in the first quadrant of the $(X_{1},X_{i})$ plane. It is obvious to notice that, because of the higher elevation and sharper slope of the asymptote compared to the hyperbola, point $Q$ has higher coordinate values than point $P$ which is the projection of the CMDFA solution vector $\vec{X}^{*}$ on $(X_{1},X_{i})$ plane i.e. $X_{1}^{up}>X_{1}^{*}$ and $X_{i}^{up}>X_{i}^{*}$. The above conclusion holds true for any projection of $\vec{X}^{*}$ on any $(X_{1},X_{i}),2\leq i\leq n$ plane. For example, the projection on $(X_{1},X_{2})$ plane will give us $X_{1}^{up}>X_{1}^{*}$ and $X_{2}^{up}>X_{2}^{*}$. Combining the outcome of all such projection for $2\leq i\leq n$ we can conclude that $\vec{X}^{up}>\vec{X}^{*}$. Which algebraically means, the intersection point among the hyperbolic cylinders in (39) and the plane in (38) is upper bounded by the intersection point among the asymptotes of the respective hyperbolic cylinders given by (46) and the plane in (38). ∎

Appendix B •

Proof of Theorem 4:.

Before we go into the business part of the proof we do some general preparatory groundwork. Using Equation (35) and the fact that we used the right root of $a_{1}$ we get,

[TABLE]

Similarly, since we are using the right roots for $a_{i},2\leq i\leq n$ we get,

[TABLE]

Equations (49) and (47) suggest that both $X_{1}^{low}$ and $X_{1}^{up}$ are decreasing functions of $\theta_{1}^{2}$. And in turn equation (37) suggests that $\mu\theta_{i}^{2},1\leq i\leq n$ are increasing functions of $\theta_{1}^{2}$. Since $\theta_{i},2\leq i\leq n$ are constants, the only thing changing in (B) is $\mu$. But $\mu$ can not increase beyond $\alpha_{1}^{2}$ because that would mean equation (34) does not have a solution. Hence, in order to increase $\theta_{1}$ as we keep on increasing the value of $\alpha_{1}$ and make it closer and closer to $1$, the value of $\mu$ also gets closer to $1$. Thus with the increament of $\theta_{1}^{2}$ the parameters given by (B) asymptotically converge to constants which we get plugging in $\mu=1$ i.e. for $2\leq i\leq n$

[TABLE]

Now that we have the groundwork done, we can proceed to prove the actual statement of the theorem.

[TABLE]

$\sum_{i=2}^{n}\frac{1-a_{i}^{low}}{1-\alpha_{i}^{2}}$ is asymptotically a constant. Equation (37) suggests $\mu^{low}$ is an increasing function of $\theta_{1}$ because $X_{1}^{low}$ is a decreasing function of $\theta_{1}$. Hence from (51),

[TABLE]

Which is an increasing function of $\theta_{1}$. Hence from (54) we see that $I^{star}-I^{low}$ is an increasing function of $\theta_{1}$. Similarly,

[TABLE]

Like the previous case we can argue that, $\sum_{i=2}^{n}\frac{1-a_{i}^{up}}{1-\alpha_{i}^{2}}$ is asymptotically a constant. Equation (37) suggests $\mu^{up}$ is an increasing function of $\theta_{1}$ because $X_{1}^{up}$ is a decreasing function of $\theta_{1}$. Hence $\frac{1-a_{1}^{up}}{1-\alpha_{1}^{2}}$ and consequently $I^{star}-I^{up}$ is an increasing function of $\theta_{1}$.

Using equations (54) and (56),

[TABLE]

where $\kappa$ is a constant. Since $a_{1}^{low}$ and $a_{1}^{up}$ are increasing functions of $\mu^{low}\theta_{1}^{2}$ and $\mu^{up}\theta_{1}^{2}$ respectively, to show $I^{up}-I^{low}$ is an increasing function of $\theta_{1}$ we need to show $\frac{\mu^{low}}{\mu^{up}}$ is an increasing function of $\theta_{1}$. Equations (49) and (47) suggest that $\frac{X_{1}^{up}}{X_{1}^{low}}$ is an increasing function of $\theta_{1}$, and in turn (37) suggests $\frac{\mu^{low}}{\mu^{up}}$ is an increasing function of $\theta_{1}$. That completes the final part of the proof. ∎