Linear-Sample Learning of Low-Rank Distributions

TL;DR

This paper establishes the sample complexity threshold for learning low-rank matrices in latent-variable models and introduces an efficient algorithm that nearly matches this lower bound, improving upon existing spectral methods.

Contribution

It determines the minimal sample size needed for learning low-rank matrices and proposes a nearly optimal, polynomial-time algorithm that advances spectral techniques.

Findings

Sample complexity lower bound: kr/^2 samples

Proposed algorithm uses kr/^2   rac{r}{}^2 \, rac{kr}{^2}\,  rac{kr}{^2}\, ext{samples}

Algorithm improves spectral methods and converges rapidly in spectral distance

Abstract

Many latent-variable applications, including community detection, collaborative filtering, genomic analysis, and NLP, model data as generated by low-rank matrices. Yet despite considerable research, except for very special cases, the number of samples required to efficiently recover the underlying matrices has not been known. We determine the onset of learning in several common latent-variable settings. For all of them, we show that learning $k\times k$, rank-$r$, matrices to normalized $L_{1}$ distance $\epsilon$ requires $\Omega(\frac{kr}{\epsilon^2})$ samples, and propose an algorithm that uses ${\cal O}(\frac{kr}{\epsilon^2}\log^2\frac r\epsilon)$ samples, a number linear in the high dimension, and nearly linear in the, typically low, rank. The algorithm improves on existing spectral techniques and runs in polynomial time. The proofs establish new results on the rapid convergence of the spectral distance between the model and observation matrices, and may be of independent interest.