Practical and Fast Momentum-Based Power Methods

TL;DR

This paper introduces two novel momentum-based power methods, DMPower and DMStream, that achieve near-optimal convergence rates with less restrictive hyperparameters, outperforming traditional methods in spectral tasks.

Contribution

The paper presents DMPower and DMStream, new momentum-based power methods that use inexact deflation to improve convergence without requiring spectral information.

Findings

DMPower outperforms vanilla power method in experiments.

Both methods match accelerated methods with perfect spectral knowledge.

Algorithms require fewer hyperparameters for convergence.

Abstract

The power method is a classical algorithm with broad applications in machine learning tasks, including streaming PCA, spectral clustering, and low-rank matrix approximation. The distilled purpose of the vanilla power method is to determine the largest eigenvalue (in absolute modulus) and its eigenvector of a matrix. A momentum-based scheme can be used to accelerate the power method, but achieving an optimal convergence rate with existing algorithms critically relies on additional spectral information that is unavailable at run-time, and sub-optimal initializations can result in divergence. In this paper, we provide a pair of novel momentum-based power methods, which we call the delayed momentum power method (DMPower) and a streaming variant, the delayed momentum streaming method (DMStream). Our methods leverage inexact deflation and are capable of achieving near-optimal convergence with far less restrictive hyperparameter requirements. We provide convergence analyses for both algorithms through the lens of perturbation theory. Further, we experimentally demonstrate that DMPower routinely outperforms the vanilla power method and that both algorithms match the convergence speed of an oracle running existing accelerated methods with perfect spectral knowledge.