Lower Bounds for the Convergence of Tensor Power Iteration on Random Overcomplete Models

TL;DR

This paper establishes that polynomially many iterations are necessary for tensor power iteration to converge in overcomplete models, challenging previous assumptions and supported by theoretical and empirical analysis.

Contribution

It provides the first lower bounds on the convergence rate of tensor power iteration in the overcomplete regime, showing polynomial iteration complexity is required.

Findings

Polynomially many steps are necessary for convergence.

Numerical experiments show successful recovery in polynomial time.

A new proof technique extends AMP analysis to tensor power iteration.

Abstract

Tensor decomposition serves as a powerful primitive in statistics and machine learning, and has numerous applications in problems such as learning latent variable models or mixture of Gaussians. In this paper, we focus on using power iteration to decompose an overcomplete random tensor. Past work studying the properties of tensor power iteration either requires a non-trivial data-independent initialization, or is restricted to the undercomplete regime. Moreover, several papers implicitly suggest that logarithmically many iterations (in terms of the input dimension) are sufficient for the power method to recover one of the tensor components. Here we present a novel analysis of the dynamics of tensor power iteration from random initialization in the overcomplete regime, where the tensor rank is much greater than its dimension. Surprisingly, we show that polynomially many steps are necessary for convergence of tensor power iteration to any of the true component, which refutes the previous conjecture. On the other hand, our numerical experiments suggest that tensor power iteration successfully recovers tensor components for a broad range of parameters in polynomial time. To further complement our empirical evidence, we prove that a popular objective function for tensor decomposition is strictly increasing along the power iteration path. Our proof is based on the Gaussian conditioning technique, which has been applied to analyze the approximate message passing (AMP) algorithm. The major ingredient of our argument is a conditioning lemma that allows us to generalize AMP-type analysis to non-proportional limit and polynomially many iterations of the power method.