Scalable and Effective Deep CCA via Soft Decorrelation

TL;DR

This paper introduces Soft CCA, a scalable deep CCA model that uses a novel stochastic decorrelation loss to improve efficiency and effectiveness in multi-view learning, outperforming existing methods.

Contribution

The paper proposes Soft CCA with a mini-batch based decorrelation loss, replacing exact decorrelation, leading to more efficient and effective deep CCA training.

Findings

Soft CCA outperforms existing deep CCA models in effectiveness.

The stochastic decorrelation loss improves training efficiency.

The SDL loss is applicable to other deep learning models.

Abstract

Recently the widely used multi-view learning model, Canonical Correlation Analysis (CCA) has been generalised to the non-linear setting via deep neural networks. Existing deep CCA models typically first decorrelate the feature dimensions of each view before the different views are maximally correlated in a common latent space. This feature decorrelation is achieved by enforcing an exact decorrelation constraint; these models are thus computationally expensive due to the matrix inversion or SVD operations required for exact decorrelation at each training iteration. Furthermore, the decorrelation step is often separated from the gradient descent based optimisation, resulting in sub-optimal solutions. We propose a novel deep CCA model Soft CCA to overcome these problems. Specifically, exact decorrelation is replaced by soft decorrelation via a mini-batch based Stochastic Decorrelation Loss (SDL) to be optimised jointly with the other training objectives. Extensive experiments show that the proposed soft CCA is more effective and efficient than existing deep CCA models. In addition, our SDL loss can be applied to other deep models beyond multi-view learning, and obtains superior performance compared to existing decorrelation losses.