A Generalized EigenGame with Extensions to Multiview Representation Learning

TL;DR

This paper introduces a novel game-theoretic approach to solving generalized eigenvalue problems, enabling scalable deep multiview representation learning with state-of-the-art results on Deep CCA.

Contribution

It develops a stochastic GEP solver inspired by Hebbian algorithms and game theory, extending to neural networks for deep multiview learning.

Findings

Achieves state-of-the-art performance on Deep CCA

Enables scalable stochastic GEP solutions with neural networks

Demonstrates effectiveness on canonical multiview datasets

Abstract

Generalized Eigenvalue Problems (GEPs) encompass a range of interesting dimensionality reduction methods. Development of efficient stochastic approaches to these problems would allow them to scale to larger datasets. Canonical Correlation Analysis (CCA) is one example of a GEP for dimensionality reduction which has found extensive use in problems with two or more views of the data. Deep learning extensions of CCA require large mini-batch sizes, and therefore large memory consumption, in the stochastic setting to achieve good performance and this has limited its application in practice. Inspired by the Generalized Hebbian Algorithm, we develop an approach to solving stochastic GEPs in which all constraints are softly enforced by Lagrange multipliers. Then by considering the integral of this Lagrangian function, its pseudo-utility, and inspired by recent formulations of Principal Components Analysis and GEPs as games with differentiable utilities, we develop a game-theory inspired approach to solving GEPs. We show that our approaches share much of the theoretical grounding of the previous Hebbian and game theoretic approaches for the linear case but our method permits extension to general function approximators like neural networks for certain GEPs for dimensionality reduction including CCA which means our method can be used for deep multiview representation learning. We demonstrate the effectiveness of our method for solving GEPs in the stochastic setting using canonical multiview datasets and demonstrate state-of-the-art performance for optimizing Deep CCA.