Multi-view Alignment and Generation in CCA via Consistent Latent Encoding

TL;DR

This paper introduces ACCA, a Bayesian approach to multi-view alignment that ensures consistent latent encodings through adversarial training, improving alignment accuracy and robustness in cross-view data analysis.

Contribution

It proposes a novel Bayesian multi-view alignment method, ACCA, which matches marginalized latent encodings to address inconsistency issues in CCA models.

Findings

ACCA achieves superior alignment in noisy settings.

The model effectively handles implicit distributions.

Experimental results show improved cross-view generation quality.

Abstract

Multi-view alignment, achieving one-to-one correspondence of multi-view inputs, is critical in many real-world multi-view applications, especially for cross-view data analysis problems. Recently, an increasing number of works study this alignment problem with Canonical Correlation Analysis (CCA). However, existing CCA models are prone to misalign the multiple views due to either the neglect of uncertainty or the inconsistent encoding of the multiple views. To tackle these two issues, this paper studies multi-view alignment from the Bayesian perspective. Delving into the impairments of inconsistent encodings, we propose to recover correspondence of the multi-view inputs by matching the marginalization of the joint distribution of multi-view random variables under different forms of factorization. To realize our design, we present Adversarial CCA (ACCA) which achieves consistent latent encodings by matching the marginalized latent encodings through the adversarial training paradigm. Our analysis based on conditional mutual information reveals that ACCA is flexible for handling implicit distributions. Extensive experiments on correlation analysis and cross-view generation under noisy input settings demonstrate the superiority of our model.