Preventing Model Collapse in Deep Canonical Correlation Analysis by Noise Regularization

TL;DR

This paper introduces NR-DCCA, a noise regularization method that prevents model collapse in Deep Canonical Correlation Analysis, ensuring stable and consistent multi-view representation learning.

Contribution

The paper proposes a novel noise regularization technique for DCCA that theoretically prevents model collapse and can be applied to other DCCA variants.

Findings

NR-DCCA outperforms baseline methods in synthetic and real datasets

The noise regularization enforces the correlation invariant property

The approach is generalizable to other DCCA-based models

Abstract

Multi-View Representation Learning (MVRL) aims to learn a unified representation of an object from multi-view data. Deep Canonical Correlation Analysis (DCCA) and its variants share simple formulations and demonstrate state-of-the-art performance. However, with extensive experiments, we observe the issue of model collapse, {\em i.e.}, the performance of DCCA-based methods will drop drastically when training proceeds. The model collapse issue could significantly hinder the wide adoption of DCCA-based methods because it is challenging to decide when to early stop. To this end, we develop NR-DCCA, which is equipped with a novel noise regularization approach to prevent model collapse. Theoretical analysis shows that the Correlation Invariant Property is the key to preventing model collapse, and our noise regularization forces the neural network to possess such a property. A framework to construct synthetic data with different common and complementary information is also developed to compare MVRL methods comprehensively. The developed NR-DCCA outperforms baselines stably and consistently in both synthetic and real-world datasets, and the proposed noise regularization approach can also be generalized to other DCCA-based methods such as DGCCA.