The Noisy Power Method: A Meta Algorithm with Applications

TL;DR

This paper introduces a robust convergence analysis of the noisy power method, a meta-algorithm applicable to various machine learning tasks involving noisy matrix computations.

Contribution

It provides a unified convergence analysis for the noisy power method, resolving open problems in streaming PCA and privacy-preserving spectral analysis.

Findings

Convergence bounds hold under significant noise levels.

The analysis subsumes previous ad-hoc bounds.

Applications include matrix completion and privacy-preserving spectral analysis.

Abstract

We provide a new robust convergence analysis of the well-known power method for computing the dominant singular vectors of a matrix that we call the noisy power method. Our result characterizes the convergence behavior of the algorithm when a significant amount noise is introduced after each matrix-vector multiplication. The noisy power method can be seen as a meta-algorithm that has recently found a number of important applications in a broad range of machine learning problems including alternating minimization for matrix completion, streaming principal component analysis (PCA), and privacy-preserving spectral analysis. Our general analysis subsumes several existing ad-hoc convergence bounds and resolves a number of open problems in multiple applications including streaming PCA and privacy-preserving singular vector computation.