The Symmetric Generalized Eigenvalue Problem as a Nash Equilibrium

TL;DR

This paper introduces a game-theoretic approach to solve the symmetric generalized eigenvalue problem efficiently, enabling scalable analysis of high-dimensional data and neural network activations.

Contribution

It formulates SGEP as a Nash equilibrium problem and develops a parallelizable algorithm with reduced complexity, improving scalability over existing methods.

Findings

Algorithm converges to Nash equilibrium

Achieves $O(dk)$ runtime complexity

Successfully applied to large-scale neural data

Abstract

The symmetric generalized eigenvalue problem (SGEP) is a fundamental concept in numerical linear algebra. It captures the solution of many classical machine learning problems such as canonical correlation analysis, independent components analysis, partial least squares, linear discriminant analysis, principal components and others. Despite this, most general solvers are prohibitively expensive when dealing with streaming data sets (i.e., minibatches) and research has instead concentrated on finding efficient solutions to specific problem instances. In this work, we develop a game-theoretic formulation of the top-$k$ SGEP whose Nash equilibrium is the set of generalized eigenvectors. We also present a parallelizable algorithm with guaranteed asymptotic convergence to the Nash. Current state-of-the-art methods require $O(d^2k)$ runtime complexity per iteration which is prohibitively expensive when the number of dimensions ($d$) is large. We show how to modify this parallel approach to achieve $O(dk)$ runtime complexity. Empirically we demonstrate that this resulting algorithm is able to solve a variety of SGEP problem instances including a large-scale analysis of neural network activations.