Dropping Convexity for More Efficient and Scalable Online Multiview Learning

TL;DR

This paper introduces a nonconvex approach to multiview representation learning, demonstrating that simple stochastic gradient descent can efficiently find global optima, supported by theoretical convergence analysis and numerical experiments.

Contribution

It proposes a nonconvex formulation for multiview learning and provides theoretical analysis showing convergence to global optima using diffusion approximations.

Findings

SGD efficiently finds global optima in the nonconvex formulation.

Theoretical convergence rates are established for the proposed method.

Numerical experiments support the theoretical results.

Abstract

Multiview representation learning is very popular for latent factor analysis. It naturally arises in many data analysis, machine learning, and information retrieval applications to model dependent structures among multiple data sources. For computational convenience, existing approaches usually formulate the multiview representation learning as convex optimization problems, where global optima can be obtained by certain algorithms in polynomial time. However, many pieces of evidence have corroborated that heuristic nonconvex approaches also have good empirical computational performance and convergence to the global optima, although there is a lack of theoretical justification. Such a gap between theory and practice motivates us to study a nonconvex formulation for multiview representation learning, which can be efficiently solved by a simple stochastic gradient descent (SGD) algorithm. We first illustrate the geometry of the nonconvex formulation; Then, we establish asymptotic global rates of convergence to the global optima by diffusion approximations. Numerical experiments are provided to support our theory.