Analyzing Tensor Power Method Dynamics in Overcomplete Regime

TL;DR

This paper analyzes the dynamics of tensor power iteration in the overcomplete regime, demonstrating its effectiveness for learning latent variable models with many components, even in noisy settings.

Contribution

It provides the first rigorous analysis of tensor power methods in the overcomplete setting, showing they can recover components with bounded error under mild conditions.

Findings

Power iteration recovers tensor components with bounded error in overcomplete regime.

Successful learning of latent variable models when the number of components exceeds data dimension.

Analysis extends spectral methods applicability to noisy, overcomplete tensors.

Abstract

We present a novel analysis of the dynamics of tensor power iterations in the overcomplete regime where the tensor CP rank is larger than the input dimension. Finding the CP decomposition of an overcomplete tensor is NP-hard in general. We consider the case where the tensor components are randomly drawn, and show that the simple power iteration recovers the components with bounded error under mild initialization conditions. We apply our analysis to unsupervised learning of latent variable models, such as multi-view mixture models and spherical Gaussian mixtures. Given the third order moment tensor, we learn the parameters using tensor power iterations. We prove it can correctly learn the model parameters when the number of hidden components $k$ is much larger than the data dimension $d$, up to $k = o(d^{1.5})$. We initialize the power iterations with data samples and prove its success under mild conditions on the signal-to-noise ratio of the samples. Our analysis significantly expands the class of latent variable models where spectral methods are applicable. Our analysis also deals with noise in the input tensor leading to sample complexity result in the application to learning latent variable models.