Probabilistic CCA with Implicit Distributions

TL;DR

This paper introduces a probabilistic CCA framework based on implicit distributions, utilizing Conditional Mutual Information and adversarial inference to better capture nonlinear dependencies in multi-view data.

Contribution

It proposes a novel probabilistic CCA model with implicit distributions, leveraging CMI and adversarial inference for improved multi-view data alignment.

Findings

Outperforms existing methods in nonlinear correlation analysis

Effective in cross-view data generation tasks

Achieves superior multi-view data alignment

Abstract

Canonical Correlation Analysis (CCA) is a classic technique for multi-view data analysis. To overcome the deficiency of linear correlation in practical multi-view learning tasks, various CCA variants were proposed to capture nonlinear dependency. However, it is non-trivial to have an in-principle understanding of these variants due to their inherent restrictive assumption on the data and latent code distributions. Although some works have studied probabilistic interpretation for CCA, these models still require the explicit form of the distributions to achieve a tractable solution for the inference. In this work, we study probabilistic interpretation for CCA based on implicit distributions. We present Conditional Mutual Information (CMI) as a new criterion for CCA to consider both linear and nonlinear dependency for arbitrarily distributed data. To eliminate direct estimation for CMI, in which explicit form of the distributions is still required, we derive an objective which can provide an estimation for CMI with efficient inference methods. To facilitate Bayesian inference of multi-view analysis, we propose Adversarial CCA (ACCA), which achieves consistent encoding for multi-view data with the consistent constraint imposed on the marginalization of the implicit posteriors. Such a model would achieve superiority in the alignment of the multi-view data with implicit distributions. It is interesting to note that most of the existing CCA variants can be connected with our proposed CCA model by assigning specific form for the posterior and likelihood distributions. Extensive experiments on nonlinear correlation analysis and cross-view generation on benchmark and real-world datasets demonstrate the superiority of our model.

Tables

Table 1. Table 1: Comparison of existing CCA variants. Linear methods are marked with grey, while others are nonlinear extensions; our method is marked with blue. The column 5 indicates whether the method avoids Gaussian distribution assumption on p ​ ( z ) 𝑝 𝑧 p({z}) . The column 6-8 indicates whether the generative models can handle corresponding distribution with implicit form.

Methods	Nonlinear		Generative	Avoids Gaussian distribution on $p\left(𝐳\right)$	Implicit
	Mapping	Criteria			$p\left(𝐳\right)$	$p(𝐱,𝐲|𝐳)$	$p\left(𝐳|𝐱,𝐲\right)$
CCA	✗	✗	✗	✗	✗	✗	✗
PCCA	✗	✗	✓	✗	✗	✗	✗
KCCA	✓	✗	✗	✗	✗	✗	✗
DCCA	✓	✗	✗	✗	✗	✗	✗
CIA	✗	✓	✗	✓	✗	✗	✗
LSCDA	✗	✓	✗	✓	✗	✗	✗
MVAE	✓	-	✗	✗	✗	✗	✗
VCCA	✓	-	✓	✗	✗	✓	✗
Bi-VCCA	✓	-	✓	✗	✗	✓	✗
ACCA	✓	✓	✓	✓	✓	✓	✓

Table 2. Table 2: Details of the datasets and network settings.

Dataset	Statistics	Dimension of $z$	Network setting (MLPs) $\hat{D}=\{1024,1024,1024\}$	Parameters
Toy dataset (Simulated)	# Tr= 8,000 # Te= 2,000	d = 10	$E_x=\{1024,1024\}$; $E_{xy}=\{1024,1024\}$; $E_y=\{1024,1024\}$	For all the dataset: learning rate = 0.001, epoch = 100. For each dataset: batch size tuned over $\{32,128,256,500,512,1000\}$; $d$ tuned over $\{10,30,50,70\}$
MNIST L/R halved dataset (MNIST_LR) (Andrew et al, 2013)	# Tr= 60,000 # Te= 10,000	d = 30	$E_x=\{2308,1024,1024\}$; $E_{xy}=\{3916,1024,1024\}$; $E_y=\{1608,1024,1024\}$
MNIST noisy dataset (MNIST_Noisy) (Wang et al, 2016)	# Tr= 60,000 # Te= 10,000	d = 50	$E_x=\{1024,1024,1024\}$; $E_{xy}=\{1024,1024,1024\}$; $E_x=\{1024,1024,1024\}$
Wisconsin X-ray Microbeam Database (XRMB) (Wang et al, 2016)	# Tr= 1.4M # Te= 0.1M	d = 112	$E_x=\{1811,1811\}$; $E_{xy}=\{1280,1280,1280\}$; $E_y=\{3091,3091\}$

Table 3. Table 3: The results of correlation analysis (best in bold).

Metric	Datasets	CCA	PCCA	MVAE	Bi-VCCA	ACCA_NoCV	ACCA (G)	ACCA (GM)
nHSIC (linear kernel)	toy	0.0010	0.1037	0.1428	0.1035	0. 8563	0.7296	0.9595
	MNIST_LR	0.4210	0.3777	0.2500	0.4612	0.5233	0.5423	0.6823
	MNIST_Noisy	0.0817	0.1037	0.4089	0.1912	0.3343	0.3285	0.4133
	XRMB	0.1735	0.2031	0.3465	0.2049	0.2537	0.2703	0.3482
nHSIC (RBF kernel)	toy	0.0029	0.2037	0.2358	0.2543	0.8737	0.5870	0.8764
	MNIST_LR	0.4416	0.3568	0.1499	0.3804	0.5799	0.6318	0.7387
	MNIST_Noisy	0.0948	0.0993	0.4133	0.2076	0.2697	0.3099	0.4326
	XRMB	0.0022	0.0025	0.0022	0.0027	0.0031	0.0044	0.0058

Table 4. Table 4: Pixel-level accuracy for full image recovery with masked inputs for different input views on the MNIST dataset.

Input (halved image)	Methods	Gray color overlaid
		1 quadrant	2 quadrants	3 quadrants
Left	MVAE	64.94	61.81	56.15
	Bi-VCCA	73.14	69.29	63.05
	ACCA	77.66	72.91	67.08
Right	MVAE	73.57	67.57	59.69
	Bi-VCCA	75.66	69.72	65.52
	ACCA	80.16	74.60	66.80

Table 5. Table 5: Details of the cross-view generation on CelebA dataset.

Dataset	Statistics	Dimension of $z$	Architecture (Conv all with batch normalization before LReLU)		Parameters
CelebA (Liu et al, 2015)	# Tr= 201,599 # Te= 1,000	d = 100	Encoders: Conv 128*5*5 (stride 2), Conv 256*5*5 (stride 2), Conv 512*5*5 (stride 2); dense: 100. Decoders: dense: 8192, relu; dConv: 512*5*5 (stride 2), dConv: 256*5*5 (stride 2), dConv: 128*5*5 (stride 2), dConv: 3*2*5, (stride1*2); Tanh.	Discriminator: $\hat{D}$: dense: 128$\to$64$\to$1, sigmoid.	Epoch = 10; Batchsize = 64; lr= 0.0002; Beta1 = 0.05;

Equations

$$p(x,y|z)=p(x|z)p(y|z).$$

$$\max \rho =corr\{W_x^{{}^{\prime }}X,W_y^{{}^{\prime }}Y\}=\frac{W_x^{{}^{\prime }}{\mathbf{\Sigma }}_{xy}{W}_y}{\sqrt{W_x^{{}^{\prime }}{\mathbf{\Sigma }}_{xx}{W}_x{W}_y^{{}^{\prime }}{\mathbf{\Sigma }}_{yy}{W}_y}},$$

$$z\sim \mathcal{N}(0,I_d),$$

$$x|z\sim \mathcal{N}(W_{x}z+{\mu }_x,{\Phi }_x),$$

$$y|z\sim \mathcal{N}(W_{y}z+{\mu }_y,{\Phi }_y),$$

$$I(X;Y|Z)=\iiint p(z)p(x,y|z)\log \frac{p(x,y|z)}{p(x|z)p(y|z)}dzdxdy$$

$$I(X;Y|Z)=I(X,Y)-I(Z_x,Z_y)=-\frac{1}{2}\log \frac{(1-{\rho }_{XY}^2)}{(1-{\rho }_{Z_x{Z}_y}^2)},$$

$$I_{\theta }(X;Y|Z)$$

$$\underset{\theta }{\min }{\mathbb{E}}_{p(x,y)}[F(\theta ;x,y)]\simeq \frac{1}{N}\sum _{i=1}^{N}F(\theta ;x^{(i)},y^{(i)}).$$

$$\min \frac{1}{N}\sum _{i=1}^{N}F(\theta ,\varphi ;x^{(i)},y^{(i)})$$

$$=-\frac{1}{N}\sum _{i=1}^N{\mathbb{E}}_{q_{\varphi }(z|x,y)}[\log p_{\theta }(x^{(i)}|z)+\log p_{\theta }(y^{(i)}|z)]+D_{KL}(q_{\varphi }(z|x,y)\parallel p_{\theta }(z)),$$

$$q_x(z)=\int q(z|x)p(x)dx,q_y(z)=\int q(z|y)p(y)dy,$$

$$q_{x,y}(z)=\iint q(z|x,y)p(x,y)dxdy,$$

$$q_x(z)\approx q_y(z)\approx q_{x,y}(z)\approx p(z).$$

$${\mathcal{L}}_{\mathrm{ACCA}}(\Theta ,\Phi ;x,y)=\frac{1}{N}\sum _{i=1}^N[-{\mathbb{E}}_{q_{{\varphi }_{xy}}(z|x,y)}[\log p_{{\theta }_x}(x|z)+\log p_{{\theta }_y}(y|z)]$$

$${\mathcal{R}}_{\mathrm{GAN}}(E_x,E_y,E_{xy},\hat{D})={\mathbb{E}}_{z^{{}^{\prime }}\sim p_{(}z)}\log (\hat{D}(z^{\prime }))+{\mathbb{E}}_{z_{xy}\sim q_{x,y}(z|x,y)}\log (1-\hat{D}(z_{xy}))$$

$$+{\mathbb{E}}_{z_x\sim q_x(z|x)}\log (1-\hat{D}(z_x))+{\mathbb{E}}_{z_y\sim q_y(z|y)}\log (1-\hat{D}(z_y))$$

$$\underset{\theta ,\varphi }{\min }\frac{1}{2N}\sum _{i=1}^N{\Vert x^{(i)}-g_x(z;{\theta }_x)\Vert }^2+{\Vert y^{(i)}-g_y(z;{\theta }_y)\Vert }^2,$$

$$\min \frac{1}{N}\sum _{i=1}^N[-\frac{1}{L}\sum _{l=1}^L[\log p_{\theta }(x^{(i)}|z_l^{(i)})+\log p_{\theta }(y^{(i)}|z_l^{(i)})]+D_{KL}(q_{\varphi }(z^{(i)}|x^{(i)})\parallel p_{\theta }(z))]$$

$$s.t.z_l^{(i)}={\mu }^{(i)}+{\Sigma }^{(i)}{ϵ}_l,\text{where}{ϵ}_l\sim \mathcal{N}(0,I_d),l=1,\dots ,L.$$

$$\min \sum _{i=1}^N[\frac{\lambda }{N}[-\frac{1}{L}\sum _{l=1}^L[\log p_{\theta }(x^{(i)}|z_{l_x}^{(i)})+\log p_{\theta }(y^{(i)}|z_{l_x}^{(i)})]+D_{KL}(q_{\varphi }(z^{(i)}|x^{(i)})\parallel p_{\theta }(z))]$$

$$+\frac{1-\lambda }{N}[-\frac{1}{L}\sum _{l=1}^L[\log p_{\theta }(x^{(i)}|z_{l_y}^{(i)})+\log p_{\theta }(y^{(i)}|z_{l_y}^{(i)})]+D_{KL}(q_{\varphi }(z^{(i)}|y^{(i)})\parallel p_{\theta }(z))]]$$

$$s.t.z_{l_x}^{(i)}={{\mu }_x}^{(i)}+{\Sigma }_x^{(i)}{ϵ}_l,z_{l_y}^{(i)}={\mu }_y^{(i)}+{\Sigma }_y^{(i)}{ϵ}_l,\text{where}{ϵ}_l\sim \mathcal{N}(0,I_d),l=1,\dots ,L,$$

$$z\sim p(z)=0.2\ast \mathcal{N}(0,1)+0.5\ast \mathcal{N}(8,2)+0.3\ast \mathcal{N}(3,1.5).$$

$$\delta =\frac{{\sum }_1^N\psi (z_x,z_y)}{N\ast \Psi },$$

Full text

1 Introduction

Canonical Correlation Analysis (CCA) (Hotelling, 1936) is a ubiquitous multi-view data analysis technique that shows promising performance in a wide range of domains, including bioinformatics (Gumus et al, 2012; Naylor et al, 2010), computer vision (Kim et al, 2007b) and natural language processing (Haghighi et al, 2008). Assuming the data to be Gaussian distributed, the classic CCA, also known as linear CCA, simply considers linear correlation for linearly transformed data. However, in practical problems, such as multi-sensor remote sensing (Ehlers, 1991; Suri and Reinartz, 2010) and medical image analysis (Studholme et al, 1999; Bach and Jordan, 2002), the structured data exhibits complex distributions. The nonlinear dependency that commonly exists in these data is insufficient to be captured with the linear CCA.

To address this problem, a stream of nonlinear CCA variants were proposed. However, these methods still rely on restrictive assumptions on the distribution of the data and latent codes. In general, most of these nonlinear variants study nonlinear dependency from two perspectives. The first line of work studies nonlinear mapping (Lai and Fyfe, 2000; Andrew et al, 2013; Wang et al, 2015). Nonlinear dependency is captured by preserving linear correlations for nonlinear transformed data. However, the linear correlation criterion can only provide complete description for the associations when the projected data are Gaussian distributed. Considering the case shown in Fig. 1.(a), the distribution of the projected data is unlikely to be Gaussian, considering the exhibited complex structure. The other works seek to higher-order dependency metrics (Vestergaard and Nielsen, 2015; Karasuyama and Sugiyama, 2012). Nonlinear dependency is achieved by capturing nonlinear dependency for linear transformed data. For these works, although the prior for the latent space is no longer limited to be Gaussian, the explicit form of the prior is still required to estimate the adopted nonlinear criterion (Vestergaard and Nielsen, 2015). The estimation would be extremely complicated for the high dimensional data (Karasuyama and Sugiyama, 2012). There are also models that imitate nonlinear CCA by shared subspace learning via nonlinear mapping (Ngiam et al, 2011). However, the data are still assumed to be Gaussian distributed. The biased assumptions adopted in existing methods make it non-trivial to comprehend CCA from a unified perspective.

To this end, this paper will systematically study probabilistic interpretation for CCA based on implicit distributions. Although probabilistic interpretation for CCA has been studied in some works, they still require the explicit form of the distributions to achieve a tractable solution for the inference. Specifically, let ${x}\in\mathbb{R}^{d_{x}}$, ${y}\in\mathbb{R}^{d_{y}}$ be the random vectors in each view, and ${z}\in\mathbb{R}^{d_{z}}$ denotes the latent variable. Probabilistic CCA (PCCA) (Bach and Jordan, 2005) provides probabilistic interpretation for linear CCA with tractable solution, due to the assumptions it inherited from linear CCA: 1): the latent codes ${z}$ follows Gaussian distribution; 2): the data in each view are transformed through linear projection. Two favorable conditions are satisfied with these assumptions: 1). $p({x},{y}|{z})$ can be modeled with the joint covariance matrix, with which the conditional independent constraint for CCA (Drton et al, 2008) can be easily imposed.

[TABLE]

2). Due to the conjugacy of the prior and the likelihood, the posterior, i.e. $p{({z}|{x},{y})}=\frac{p{({x},{y}|{z})}p({{z}})}{p{({x},{y})}}$ can be presented with an analytic solution (Tipping and Bishop, 1999). These two conditions facilitate a valid probabilistic interpretation for linear CCA. Probabilistic interpretation for nonlinear CCA models is much more difficult to achieve, since both the two conditions are violated. Specifically, as nonlinear dependency is to be captured, the Gaussian assumption made on the prior is violated. Therefore, the linear correlation is no longer an ideal criterion for the problem. Furthermore, as the mappings to the latent space are complex in some nonlinear variants, conjugacy between the prior and the likelihood is violated. The intractable posterior distributions make it hard to enforce the conditional independent constraint (Eq. (1)) for the problem. Although deep variational CCA (VCCA) (Wang et al, 2016) is proposed to interpret nonlinear CCA, it still adopts a Gaussian distribution assumption on the posterior to achieve a tractable solution for the inference of multi-view data with KL-divergence. Besides, as there is no explicit connection between its objective and that of CCA, the relevancy of VCCA and CCA is still unclear.

Considering the aforementioned challenges, in this paper, we provide probabilistic interpretation for CCA with Conditional Mutual Information (CMI). We present minimum CMI as a new criterion for CCA to consider both linear and nonlinear dependency. To estimate CMI without the explicit form of the distributions, we derive an objective which can provide an estimation for CMI with inference methods. To facilitate efficient Bayesian inference for multi-view analysis with implicit distributions, we propose Adversarial CCA (ACCA), which achieves consistent encoding for multi-view data with the consistent constraint imposed on the marginalization of the implicit posteriors. Most of the existing CCA variants can be connected with our model based on certain assumptions on the posterior and likelihood distributions. We demonstrate the superior consistency achieved with our model contributes to better nonlinear correlation analysis and cross-view generation performance on both benchmark and real-world datasets. The contributions of this work can be summarized as follows:

We present minimum Conditional Mutual Information (CMI) as a new criterion for CCA, which can consider both linear and nonlinear dependency for CCA without additional concern for the conditional independent constraint.

• 2.

We provide an objective which can provide an estimation for CMI in multi-view learning, without the explicit form of the distributions.

• 3.

We propose Adversarial CCA (ACCA) which facilitates Bayesian inference for multi-view analysis with implicit distributions. Most of the existing CCA variants can be explained with our model with specific distribution assumptions.

• 4.

Achieving superior consistency for the encoding of multi-view data, our proposed ACCA presents superior performance in nonlinear dependency analysis and cross-view data generation task.

The remainder of this paper is organized as follows. Section 2 discusses the related works and preliminaries of our study. In Section 3, we present Conditional Mutual Information as a new criterion for CCA and provide the objective for efficient estimation for CMI. Section 4 presents our design of Adversarial CCA (ACCA) and explains its connection with existing CCA variants. Section 5 demonstrates the superiority of our model through empirical results on both synthetic and real-world datasets. Section 6 concludes the paper and envisions the future work.

2 Related work and preliminaries

In this section, we give a review of the existing works that are related to our study.

Canonical Correlation Analysis (CCA): CCA (Hotelling, 1936) is a powerful statistic tool for multi-view data analysis. Let $X=\{x^{(i)},y^{(i)}\}_{i=1}^{N}$ consists $N$ i.i.d. samples with pairwise correspondence in multi-view scenario, the classic CCA aims to find linear projections for the two views, ($W_{x}^{{}^{\prime}}X,W_{y}^{{}^{\prime}}Y$), such that the linear correlation between the projections are mutually maximized, namely

[TABLE]

where $\rho$ denotes the correlation coefficient, ${\mathbf{\Sigma}_{xx}}$ and ${\mathbf{\Sigma}}_{yy}$ are the covariance of $X$ and $Y$, respectively, ${\mathbf{\Sigma}}_{xy}$ denotes the cross-covariance of $X$ and $Y$.

Assuming the data to be Gaussian distributed, the classic CCA simply captures linear correlation under linear projections. It is often insufficient to analyse complex real-world data that exhibit higher-order dependency (Suzuki and Sugiyama, 2010).

Nonlinear CCA variants: Various nonlinear CCA variants were proposed to capture nonlinear dependency in multi-view problems. Most of these models can be grouped into two categories according to the extension strategy (see Table 1). For the first category, the nonlinear extension is conducted by capturing linear correlation for nonlinear transformed data, e.g. Kernel CCA (KCCA) (Lai and Fyfe, 2000) and Deep CCA (DCCA) (Andrew et al, 2013). The adopted linear correlation is optimal only when the common latent space is Gaussian distributed. However, since nonlinear mappings are adopted, the posterior is intractable, in which the Gaussian distribution assumption can not be fulfilled. For the other category, the extension is conducted by capturing high-order dependency for linear transformed data. Most of the works adopt mutual information or its variants as the nonlinear dependency measurement, e.g. Canonical Information Analysis (CIA) (Vestergaard and Nielsen, 2015) and Least-Squares Canonical Dependency Analysis (LSCDA) (Karasuyama and Sugiyama, 2012), respectively. However, the explicit form of the latent distribution is required to estimate the adopted criterion in these methods. The estimation would be extremely complicated for the high dimensional data (Kraskov et al, 2004). There are also works that adopt Multi-View AutoEncoders (MVAE) (Ngiam et al, 2011) to discover the correlation of the multi-view data via multi-view reconstruction. However, it still assumes the latent space to be Gaussian distributed, which is often violated practically (see Fig. 1.(b)). Furthermore, since there is no explicit objective for the model as discovering correlations, its connection with CCA is unclear.

Probabilistic interpretation for CCA: To deepen the understanding of CCA-based models, some works attempt to study a probabilistic interpretation for CCA.

PCCA (Bach and Jordan, 2005) studies the probabilistic interpretation for linear CCA (Eq. (2)). It states that maximum likelihood estimation for the model in Fig. 2.(a), leads to the canonical correlation directions, by

[TABLE]

where $d$ denotes the dimension of the projected space. Obviously, inheriting from the linear CCA, PCCA adopts two assumptions: 1). the latent space follows Gaussian distribution, with which the linear correlation is a ideal criterion; 2). the data in each view are transformed through linear projection. As the prior and the likelihood are conjugate, $p({x},{y}|{z})$ can be modeled with the joint covariance matrix, with which the conditional independent constraint (Drton et al, 2008) can be easily imposed.

However, there lacks a probabilistic interpretation for nonlinear CCA variants that can explain the nonlinear dependency captured by these models. An intuitive idea is to generalize PCCA to these nonlinear models. However, PCCA cannot interpret nonlinear CCA variants for mainly two reasons: 1). The correlation criterion adopted in PCCA cannot capture the high-order dependency between the variables. 2). The conditional independent constraint is hard to enforce in the optimization of nonlinear CCA variants. In (Wang et al, 2016), deep variational CCA (VCCA) is presented, which is claimed to provide interpretation for the first category of nonlinear CCA models parametrized by DNNs. In addition, a variant named bi-VCCA is further presented, which incorporates encoding for both the two views. However, the connection between their objectives and nonlinear dependency is not clear. Furthermore, the prior $p({z})$ and the approximate posteriors $q({z}|{x},{y})$ are still required to be Gaussian distributions so that the KL-divergence can be computed analytically. Since data are usually complex in real-world tasks, simple Gaussian prior restricts the expressive power of these CCA models (Mescheder et al, 2017).

Consequently, there are two challenges to consider probabilistic interpretation for nonlinear CCA models. First, a new criterion which can measure nonlinear dependency with implicit distributions is required. Second, the conditional independent constraint needs to be easily imposed with the proposed criterion.

3 Probabilistic interpretation for CCA with CMI

In this section, we present a probabilistic interpretation for CCA with Conditional Mutual Information (CMI). First, we present the minimum CMI as a new criterion for CCA in Sect. 3.1. Then, we provide our objective for the estimation of CMI in Sect. 3.2.

3.1 Minimum CMI: a new criterion for CCA

Given three random variables ${X},{Y}$ and ${Z}$, the CMI measures conditional dependency, i.e. mutual information between ${X}$ and ${Y}$ given the variable ${Z}$ (Zhang et al, 2014).

[TABLE]

In general, CMI owns two properties:

• missing1 ;\nodeat (char.center) ;

It is a general measurement that can evaluate both linear and nonlinear dependency between the variables (Bertran et al, 2019).

• missing2 ;\nodeat (char.center) ;

$I({X};{Y}|{Z})\geq 0$ (Cover and Thomas, 2012).

The connection between CMI and the general objective of CCA can be illustrated with Fig. 3. Specifically, from the probabilistic perspective, given the multi-view data (${X},{Y}$), CCA aims to find latent variable ${Z}$ with which the dependency between ${X}$ and ${Y}$ can be maximumly captured. Adopting mutual information as the dependency measurement, the mutual information between the latent codes ${Z}_{x}$ and ${Z}_{y}$ is to be maximized. Since $I{({X};{Y})}$ is constant for a given problem, $I{({X};{Y}|{Z})}$ is to be minimized for CCA. This makes the minimum CMI a new criterion for CCA models.

The proposed minimum CMI criterion overcomes the challenges concluded in Sect. 2 simultaneously. First, CMI can measure the nonlinear dependency (Rahimzamani and Kannan, 2017) between two random variables, making it a competent statistic to interpret nonlinear CCA variants. Second, the conditional independent constraint for CCA-based models is automatically satisfied with the minimum CMI, since the optimal, $I({X};{Y}|{Z})=0$ (Cover and Thomas, 2012), is achieved with Eq. (1).

Discussion: As shown with Fig.3, in the multi-view learning scenario, CMI can be interpreted as the reduction of mutual information between the two views when mapped into the common latent space. The superiority of minimum CMI as CCA criterion can be explained from the following two aspects:

• 1).

Compared with the linear correlation (Taylor, 1990), CMI is a more general criterion that can interpret existing CCA models without simplifying distribution assumptions.

• 2).

Compared with $I({Z}_{x},{Z}_{y})$ adopted in  (Vestergaard and Nielsen, 2015) and (Karasuyama and Sugiyama, 2012), CMI presents explicit connection with the conditional independent assumption in CCA, which makes it a better criterion to consider nonlinear dependency for CCA.

Remark 1

As a variant of mutual information, CMI considers information about all the dependency (Gelfand and Yglom, 1959), both linear and nonlinear, in CCA without explicit assumptions on the mapping and the distribution of the latent space. Existing CCA models can all be explained with this criterion as follows.

The classic linear CCA can be explained with CMI since there exists an exact relationship between linear correlation and CMI (refer to Eq.(5))when both the original data and the latent codes are assumed to be Gaussian distributed (Brillinger, 2004).

[TABLE]

where $\rho_{XY}=corr\{{X},{Y}\}$ and $\rho_{Z_{x}Z_{y}}=corr\{{Z}_{x},{Z}_{y}\}$.

The nonlinear CCA variants (listed in Table 1) can be explained with CMI, since CMI eliminates the restriction on the form of mapping and the distribution of the common latent space. $\blacksquare$

3.2 Objective of the estimation for CMI

To provide an estimation for CMI without the explicit form of the distribution, we derive an objective which can provide an estimation for CMI with efficient inference methods.

Based on the definition of CMI, i.e. Eq. (4), we have

[TABLE]

The $H(X,Y)$ is a constant and has no effect on the optimization. Consequently, the minimum CMI criterion can be achieved by minimizing the remaining terms, namely

[TABLE]

where $F(\theta;x,y)=-\mathbb{E}_{p_{\theta}({z}|{x},{y})}~{}[\log{p_{\theta}({x}|{z})}+\log{p_{\theta}({y}|{z})}]+D_{KL}(p_{\theta}({z}|{x},{y})\parallel p({z})).$

Although Eq. (6) avoids the difficulties in direct estimation CMI, the objective is still hard to optimize since the posterior $p_{\theta}(z|x,y)$ is unknown or intractable for practical multi-view learning problems. Consequently, efficient Bayes inference technique is to be adopted for an estimation for CMI based on implicit distributions.

4 Adversarial CCA

In this section, we present our proposed Adversarial CCA (ACCA), which facilitates efficient Bayesian inference for multi-view analysis based on implicit distributions. We first state the deficiency of existing generative CCA variants in cross-view analysis in Sect. 4.1. Then, we give our motivation for ACCA in Sect. 4.2. Next, we present the technical details, including the model design, the formulation and training details of ACCA in Sect. 4.3, Sect. 4.4 and Sect. 4.5 respectively. In the end, we discuss the connection between ACCA and other probabilistic CCA variants in Sect. 4.6.

4.1 Deficiency of CCA variants in cross-view analysis

As a powerful technique for multi-view data analysis, CCA and its variants have been widely adopted in discriminant learning tasks (Kim et al, 2007a; Li et al, 2018). However, seldom attention has been paid on the generative CCA model which is capable to handle cross-view data analysis task. Considering we have images of different angles or views for an object, it would be promising to generate one view of the data given single input from the other view, namely cross-view data generation (Regmi and Borji, 2018). The application would be even more interesting for multimodal data, e.g. pairwise visual and audio data. The study can also benefit the common partial view problem (Li et al, 2014; Shi et al, 2019) in multi-view data analysis tasks.

Alignment of the multi-view data plays a crucial role in the aforementioned cross-view generation task (Ngiam et al, 2011). There are two challenges to achieve superior alignment in these tasks. First, handling implicit distributions would greatly benefit the expressiveness for the encodings of each view (Mescheder et al, 2017), but how to achieve a tractable solution for implicit posterior remains a problem. Second, how to achieve consistency for the implicit posteriors obtained with each view.

Existing generative CCA variants suffer from misalignment in these tasks due to their biased assumption on the posterior distribution. Specifically, since PCCA adopts linear mapping on Gaussian distributed data, it can directly achieve an analytic solution for the posterior (discussed in Section. 1). Bi-VCCA also assumes the posterior to be Gaussian distributed to achieve tractable solutions for the inference of each view. However, it adopts a heuristic combination of KL-divergence to achieve consistency of the posteriors, which yields an improper inference for multi-view analysis. Consequently, Bi-VCCA still suffers from the misaligned encoding for the two views.

4.2 Motivation

To tackle the aforementioned challenges in the cross-view analysis tasks, we derive a tractable solution for implicit posterior based on the first principle of Bayesian inference—marginalization. Since the marginalization on $p_{\theta}({z}|{x},{y})$ in Eq.(3.2), i.e. $\iint p_{\theta}({z}|{x},{y})\,p({x},{y})d{x}d{y}$, is intractable, we introduce an approximation for the exact posterior with a tractable posterior $q(z|x,y)$ such that a tractable solution can be achieved for Eq. (6) with

[TABLE]

Consequently, we can perform marginalization on $q_{\phi}({z}|{x},{y})$ to facilitate Bayesian inference for cross-view analysis.

Obviously, this objective consists of two parts: (1). the reconstruction term, defined by expectation of the data log-likelihood of the two views; (2). the prior regularization term, defined by the KL-divergence of the approximated posterior $q_{\phi}({z}|x,y)$ and the prior $p({z})$.

Based on Eq. (7), we propose Adversarial CCA, which achieves consistent encoding for multi-view data with the consistent constraint imposed on the marginalization of the implicit posteriors.

4.3 Design of ACCA

Graphical diagram of the ACCA is depicted in Fig. 2.(d). Specifically, we adopt two schemes to achieve consistent encoding that promotes the multi-view data alignment. Holistic Encoding: To facilitate the cross-view analysis task, we provide holistic information for the inference, i.e. $q({z}|{x},{y})$, $q({z}|{x})$ and $q({z}|{y})$, in ACCA.

Remark 2

Based on Eq.(7), we first explicitly model $q({z}|{x},{y})$ by encoding an auxiliary view $XY$, which contains all the information of the two views. Then, we further adopt the two principle encodings $q({z}|{x})$ and $q({z}|{y})$ to facilitate the cross-view generation task. As holistic information is provided for the encoding of $z$, this scheme contributes to superior expressiveness of the common latent space.

Adversarial learning: We then adopt an adversarial learning scheme to achieve consistency for three encodings by matching the marginalization of the implicit posteriors.

Remark 3

Since the prior regularization term in Eq. (7) is still intractable for implicit posterior distribution, we design an adversarial learning scheme to achieve multi-view consistency by enforcing the approximation of the three marginalized posterior to the same implicit prior $p({z})$. Within this scheme, each encoder defines a marginalized posterior over ${z}$ (See Eq.(1) in (Makhzani et al, 2015)).

[TABLE]

[TABLE]

As the true posterior $p({z}|{x},{y})$ is simultaneously approximated with the three terms, we adopt an generative adversarial network (GAN) that adopts the three inference model as generators and one shared discriminator to enforce the approximation of the three marginalized posteriors.

[TABLE]

Since the three marginalized posteriors are driven to match the prior $p({z})$ simultaneously, the adversarial learning scheme provides a consistent constraint for the incorporated encodings in ACCA. $\blacksquare$

4.4 Formulation of ACCA

Based on Eq. (7) and the our design of ACCA, we formulate ACCA with the following equation and design the network structure as Fig. 4.

[TABLE]

where $\Theta$ and $\Phi$ in the left hand side denotes the parameters of the encoders and the decoders respectively, i.e. $\Theta$ = $\{\theta_{x},\theta_{y}\}$ and $\Phi$ = $\{\phi_{xy},\phi_{x},\phi_{y}\}$.

Remark 4

The framework of ACCA consists of 6 subnetworks (refer to Fig. 4). The three encoders, $\{E_{x},E_{xy},E_{y}\}$, and the two decoders $\{D_{x},D_{y}\}$ compose the view-reconstruction scheme, which corresponds to the three terms in Eq. (4.4). The three encoders also constitute a adversarial scheme with the shard discriminator $\hat{D}$. $\{E_{x},E_{xy},E_{y},\hat{D}\}$ provides a consistent constraint with $\mathcal{R}_{\rm GAN}$, namely

[TABLE]

Here we adopt $\{{z}_{{x}},{z}_{{y}},{z}_{{xy}}\}$ to highlight ${z}$ obtained with different encodings. $\blacksquare$

4.5 Training

For ACCA, we assume that each encoding, i.e. $q(z|*)$, to be a deterministic function of the corresponding view, and train the autoencoders and the adversarial networks jointly with ADAM Optimizer (Radford et al, 2015) in two phases - the reconstruction phase and the regularization phase - executed on each mini-batch. In the reconstruction phase, the autoencoders update the encoders and the decoders to minimize the reconstruction error of the inputs in the three views. In the regularization phase, the adversarial network trained with the classic mechanism presented in (Goodfellow et al, 2014), with the generators (encoders) trained to maximize $\log D(z_{*})$ instead. Once the training procedure is done, the decoders of the autoencoder will define generative models that map the imposed prior of $p(z)$ to the data distribution in each view.

4.6 Connection with other CCA variants

Facilitating Bayesian inference for implicit distributions based on marginalization in Eq. (7), our proposed ACCA can be connected with other existing CCA variants with certain assumptions on the posterior and likelihood distributions.

Example 1: PCCA (Bach and Jordan, 2005).  With an explicit conditional independent assumption for CCA, PCCA adopts Gaussian assumptions for both the likelihood and the prior, i.e. Eq. (2), to interpret linear CCA. Under the conditional independent constraint, the minimum CMI criterion, i.e. $I({X};{Y}|{Z})=0$, is naturally satisfied. Then, due to the conjugacy of the prior and the likelihood, the posterior in Eq.(3.2) can be presented with an analytic solution, with which the model parameters can be directly estimated with EM algorithm.

Example 2: MVAE (Ngiam et al, 2011). If we consider Gaussian models with ${z}\sim\mathcal{N}({\mu},{0})$, $p_{\theta}(x|z)=N(g_{x}(z;\theta_{x}),I)$ and $p_{\theta}(y|z)=N(g_{y}(z;\theta_{y}),I)$, we can see that the reconstruction term in Eq. (7) measures the $l_{2}$ reconstruction error of the two inputs from the latent code $z$ through the DNNs defined with $g_{x}$ and $g_{y}$. Without consistent constraint on the multi-view encodings, the objective of MVAE is given as

[TABLE]

Example 3: VCCA (Wang et al, 2016). Considering a model where the latent codes ${z}\sim\mathcal{N}({\mu},\Sigma)$ and the observations $x|z$ and $y|z$ both follow implicit distribution, VCCA adopts variational inference to get the approximate posterior in Eq.(7) with two additional assumptions: 1). The input view at the test time can provide sufficient information for the multi-view encoding, namely $q_{\theta}(z^{(i)}|x^{(i)},y^{(i)})\approx{q_{\theta}(z^{(i)}|x^{(i)})}$; 2). The variational approximate posterior $q_{\theta}(z^{(i)}|x^{(i)})\sim N(z^{(i)};\mu^{(i)},\Sigma^{(i)})$, where $\Sigma^{(i)}$ is a diagonal covariance, i.e. $\Sigma^{(i)}=\rm diag(\sigma_{1}^{2},\ldots,\sigma_{d_{z}}^{2})$. In this case, the KL divergence term can be explicitly computed with $D_{KL}(q_{\phi}({z}^{(i)}|{x}^{(i)})\parallel p_{\theta}({z}^{(i)}))=-\frac{1}{2}\sum_{j=1}^{d_{z}}(1+\log\sigma_{ij}^{2}-\sigma_{ij}^{2}-\mu_{ij}^{2})$. As $q_{\theta}(z^{(i)}|x^{(i)})$ is confined with an explicit form, Monte Carlo sampling (Hastings, 1970) is adopted to approximate the expected log-likelihood term in Eq.(7). Drawing $L$ samples $z_{i}^{l}\sim q_{\phi}({z}^{(i)}|{x}^{(i)})$, the objective of VCCA is given as

[TABLE]

Example 4: Bi-VCCA (Wang et al, 2016). As a variant of VCCA, Bi-VCCA adopts the encoding of both the two views, namely, $q_{\theta}({z}|{x})$ and $q_{\theta}({z}|{y})$ to solve the problem and its objective is given as a heuristic combination of Eq. (4.6) derived with each encodings.

[TABLE]

where $\lambda\in[0,1]$, which is the trade-off factor between the two encodings.

5 Experiments

In this section, we first verify that the minimum CMI criterion can be achieved with the optimization of ACCA. Next, we demonstrate the superiority of ACCA in handling implicit distributions with the prior specification. Then, we conduct correlation analysis to demonstrate the performance of ACCA in capturing nonlinear dependency. Then, we verify the consistency for the inference of multi-view data in ACCA through alignment analysis and cross-view generation.

5.1 Verification of optimizing CMI

We verify that ACCA achieves the proposed CMI criterion with one of the most commonly used multi-view learning dataset - MNIST left right halved dataset (MNIST_LR) (Andrew et al, 2013). Details about the dataset and network design are in Table 2.

We estimate CMI the model training process with an open source non-parametric Entropy Estimation toolbox111https://github.com/gregversteeg/NPEET. Fig. 5 illustrates that the CMI gradually decreases during the training of ACCA and it reaches zero at a relatively early stage in the convergence of ACCA. The trend indicates two points:

• 1).

Instantiated with our objective for CCA based on CMI, i.e. Eq. (6), ACCA implicitly minimizes CMI during the training process and the minimum CMI criterion can be achieved at the convergence of the objective.

• 2).

The conditional independent constraint (Eq. (1)) is automatically satisfied, i.e. $I{({X};{Y}|{Z})=0}$, with the objective of ACCA.

5.2 Prior Specification

We conduct correlation analysis on a toy dataset with specific prior to verify that ACCA benefits from handling implicit distributions in capturing nonlinear dependency.

Toy dataset: Analogizing to (William, 2000), we construct a toy dataset that exists nonlinear dependency between the two views for the test. Specifically, let $X=W_{1}Z$ and $Y=W_{2}Z^{2}$, where $Z$ denotes a 10-D vector with each dimension $z\sim p(z)$, and $W_{1}\in\mathbb{R}^{10*50}$ , $W_{2}\in\mathbb{R}^{10*50}$ are the random projection matrices to construct the data. Details for the setting is presented in Table 2. As we consider nonlinear dependency with non-Gaussian prior, we set $p({z})$ with a mixture of Gaussian distribution in this experiment.

[TABLE]

Dependency metric: Hilbert Schmidt Independence Criterion (HSIC) (Gretton et al, 2005) is a state-of-the-art measurement for the overall dependency among variables. In this work, we adopt the normalized estimate of HSIC(nHSIC) (Wu et al, 2018) as the metric to measure the dependency captured by the embeddings of the test set ($Z_{X_{Te}}$ and $Z_{Y_{Te}}$) of each methods. The nHSIC computed with linear kernel and RBF kernel ($\sigma$ set with the F-H distance between the points) are both reported.

Baselines: We compare ACCA with several state-of-the-art generative CCA variants.

• •

CCA (Hotelling, 1936): Linear CCA model that learns linear projections of the two views that are maximally correlated.

• •

PCCA (Bach and Jordan, 2005): Probabilistic variant of linear CCA, which yields EM updates to get the final solution.

• •

MVAE (Ngiam et al, 2011): Multi-View AutoEncoders, an CCA variant that discovers the dependency among the data via multi-view reconstruction.

• •

Bi-VCCA (Wang et al, 2016): Bi-deep Variational CCA, a representative generative nonlinear CCA model restricted with Gaussian prior.

• •

ACCA_NoCV: An variant of ACCA which is designed without the encoding of the complementary view $XY$. This is compared to verify the efficiency of the holistic encoding scheme in ACCA.

• •

ACCA(G); ACCA model implement with the standard Gaussian prior.

• •

ACCA(GM): ACCA model implement with a Gaussian mixture prior. The prior is set as the exact prior, i.e. Eq (5.2), in the experiments.

As ACCA handles implicit distributions, which benefits its ability to reveal the intricacies of the latent space, higher nonlinear dependency is expected to achieve with this method. Table 3 presents the results of our correlation analysis. The table is revealing in several ways:

• 1).

Both CCA and PCCA achieve low nHSIC value on the toy dataset, due to their insufficiency in capturing nonlinear dependency.

• 2).

The results of MVAE and Bi-VCCA are unsatisfactory. MVAE is influenced probably because it does not have the inference mechanism that qualifies the encodings. The Gaussian assumption made on the data distributions may also restrict its performance. Bi-VCCA suffers mainly because the heuristic combination of KL-divergence adopted for the approximation of the inferences causes inconsistent encoding problem.

• 3).

The three variants of ACCA all achieve good performance here. This indicates that the consistent constraint imposed on the marginalization in our model benefits the models’ ability to capture nonlinear dependency.

• 4).

ACCA (GM) archives the best result on both settings among our methods. This verifies that ACCA benefits from the ability to handle implicit distributions.

5.3 Correlation Analysis

We further test ACCA in capturing nonlinear dependency on three commonly used multi-view datasets, see (Table 2). The result is presented in Table 3. We can see that

• 1).

Both of our models, ACCA(G) and ACCA (GM) achieve superb performance. The results of ACCA are much better than that of the state-of-the-art baselines. This indicates that the consistent constraint imposed on the marginalized posterior contributes to consistent encoding for the multi-view data which benefits for capturing dependency.

• 2).

ACCA(G) achieves better results than ACCA_NoCV on most of the settings. This demonstrates that the adopted holistic encoding scheme also contributes to better dependency capturing ability for ACCA.

5.4 Verification of consistent encoding

In this subsection, we specially verify the consistent encoding of ACCA with MNIST_LR dataset. We first illustrate the approximation of the three encodings in ACCA. Then, we conduct alignment verification to verify the effect of consistent encoding.

5.4.1 Approximation of three encodings

We adopt MMD distance to verify the approximation of the three encodings in ACCA. Specially, we calculate the sum of the MMD distance between the three encodings and the prior $p(z)$ in Eq. (8) during the training process. Fig. 6 shows that result decreases during the convergence of ACCA. This verifies that ACCA achieves Eq. (9), i.e. the consistent encoding during the optimization.

5.4.2 Alignment verification

We first embed the data into two-dimensional space to verify the alignment of the multi-view data in the common latent space.

Specifically, projecting the paired testing data to the latent space with Gaussian prior, we take the origin $O$ as the reference and adopt angular difference to measure the distance of the paired embeddings, i.e. $\phi({z}_{x},{z}_{y})=\angle{z}_{x}O{z}_{y}$ (see Fig. 7.(b)). The misalignment degree of the multi-view is given by

[TABLE]

Where $\Psi$ denotes the maximum angle of all the embeddings and $N$ is the number of data pairs.

We compare ACCA with MVAE, Bi-VCCA and ACCA_NoCV here, as they have encoding for both the two views. The results are presented in Fig. 7. We have the following observations.

• 1).

The regions for the paired embeddings of Bi-VCCA are even not overlapped and the misalignment degree of Bi-VCCA is $\delta=2.3182$, which is much higher than the others. This shows that Bi-VCCA greatly suffers from the misaligned encoding problem for the multi-views due to the inferior inference of the multi-view data.

• 2).

Our two methods, ACCA and ACCA_NoCV, achieves superior alignment performance compared with MVAE (small misalignment degree) and Bi-VCCA (better-overlapped regions and small misalignment degree). This shows the effectiveness of the consistent constraint on the marginalization for view alignment in ACCA.

• 3).

The embeddings of ACCA are uniformly distributed in the latent space compared with that of ACCA_NoCV, indicating that the complementary view, $XY$ provide additional information for the holistic encoding, which benefits the effectiveness of the common latent space.

5.5 Cross-view generation

We apply ACCA to novel cross-view image generation task, where consistent encoding is critical for the performance. Specifically, this task aims at whole-image recovery given the partial input image as one of the views. We conduct the experiment on MNIST and CelebA (Liu et al, 2015), which are both commonly used for image generation. We add noise to the original MNIST to verify the robustness of ACCA. Specifically, we divide the test data in each view into four quadrants and masked one, two or three quadrants of the input with grey color (Sohn et al, 2015) and use the noisy images as the input for test. The result is evaluated from both qualitative and quantitative aspects. For CelebA, we halved the RGB images into top-half and bottom-half and designed CNN architecture for this problem. Details for the network design is given in Table. 5. The evaluation is conducted on the quality of the generated image, e.g. is the image blurred, does the image shows clear misalignment at the junctions in the middle. Since MVAE and Bi-VCCA are the baseline methods that can support this cross-view generation task, we compare the two methods here.

5.5.1 MNIST

Qualitative analysis: Fig. 8 presents some of the recovered images (column 3-5) obtained with 1 quadrant input. This figure clearly illustrates that, given the noisy input, the images generated with ACCA is more real and recognizable than that of MVAE and Bi-VCCA.

• 1).

The image generated with MVAE shows the worst quality. The images contain much noise compared with other methods. In many cases, the ”digit” is hard to identify, e.g. case (b). In addition, the generated image of MVAE shows clear misalignment at the junctions of the halved images, e.g. case (a).

• 1).

The images generated by Bi-VCCA are much more blurred and less recognizable than that of ACCA, especially in case (a) and case (b).

• 2).

ACCA can successfully recover the noisy half images which are even confusing for our human to recognize. For example, in case (b), the left-half image of digit “5” looks similar to the digit “4”, ACCA succeeds in recovering the true digit.

Quantitative evidence: We compare the pixel-level accuracy of the generated images in Table 4. It shows that our ACCA consistently outperforms Bi-VCCA given the different level of masked input images. It is also interesting that using the left-half images as the input tends to generate better images than using the right-half. It might be because of the right-half images contain more information than the left-half images, which will result in better network training for more accurate image generation.

5.5.2 CelebA

Fig. 9 shows the image samples generated for CelebA dataset. We have mainly two observations.

• 1).

The samples generated with MVAE shows clear misalignment at the junctions, especially when the images are with colored backgrounds. Some of the images are much blurred to see the details, e.g. the samples circled with red.

• 2).

The samples generated by Bi-VCCA are commonly blurred than the other two. The observation is quite obvious in the image generated with the top-half image, which contains much less details than the bottom-half image.

• 3).

The images generated with ACCA show better quality compared with the others, considering both the clarity and the alignment of junctions.

6 Conclusion

In this paper, we present a probabilistic interpretation for CCA based on implicit distributions. Our study discusses the restrictive assumptions of existing CCA variants and provides a unified probabilistic interpretation for these models based on Conditional Mutual Information. We present minimum CMI as a new criterion for CCA and provides an objective which can provide efficient estimation for CMI without explicit distributions. We further propose Adversarial CCA which shows superior performance in both nonlinear correlation analysis and cross-view generation, due to the consistent constraint imposed on marginalizations. Due to the matching of multiple encodings, ACCA can also be adopted to other practical tasks, such as image captioning and translation. Furthermore, because of the flexible architecture designed in Eq.(4.4), proposed ACCA can be easily extended to multi-view task of $n$ views, with ($n+1$) encoders and ($n$) decoders.