Dynamically-Scaled Deep Canonical Correlation Analysis

TL;DR

This paper introduces a dynamic scaling approach for deep CCA models, enabling input-dependent parameterization that enhances the correlation of learned representations and improves retrieval performance.

Contribution

It proposes a novel input-dependent parameterization method for deep CCA, improving the correlation of learned features over traditional fixed-parameter models.

Findings

More correlated representations than conventional models

Improved retrieval results on multiple datasets

Input-dependent scaling enhances model capacity

Abstract

Canonical Correlation Analysis (CCA) is a method for feature extraction of two views by finding maximally correlated linear projections of them. Several variants of CCA have been introduced in the literature, in particular, variants based on deep neural networks for learning highly correlated nonlinear transformations of two views. As these models are parameterized conventionally, their learnable parameters remain independent of the inputs after the training process, which may limit their capacity for learning highly correlated representations. We introduce a novel dynamic scaling method for training an input-dependent canonical correlation model. In our deep-CCA models, the parameters of the last layer are scaled by a second neural network that is conditioned on the model's input, resulting in a parameterization that is dependent on the input samples. We evaluate our model on multiple datasets and demonstrate that the learned representations are more correlated in comparison to the conventionally-parameterized CCA-based models and also obtain preferable retrieval results. Our code is available at https://github.com/tomerfr/DynamicallyScaledDeepCCA.