Spectral Experts for Estimating Mixtures of Linear Regressions

TL;DR

This paper introduces a new efficient and provably consistent method for estimating mixtures of linear regressions using tensor decomposition, overcoming local optima issues of traditional EM algorithms.

Contribution

It develops a novel tensor-based estimator with theoretical convergence guarantees for mixture of linear regressions, improving over existing local optimization methods.

Findings

The estimator achieves provable convergence rates.

It outperforms EM in empirical evaluations.

The method efficiently recovers model parameters.

Abstract

Discriminative latent-variable models are typically learned using EM or gradient-based optimization, which suffer from local optima. In this paper, we develop a new computationally efficient and provably consistent estimator for a mixture of linear regressions, a simple instance of a discriminative latent-variable model. Our approach relies on a low-rank linear regression to recover a symmetric tensor, which can be factorized into the parameters using a tensor power method. We prove rates of convergence for our estimator and provide an empirical evaluation illustrating its strengths relative to local optimization (EM).