Unveiling the Cycloid Trajectory of EM Iterations in Mixed Linear Regression

TL;DR

This paper analyzes the iterative behavior and convergence of the EM algorithm in mixed linear regression, revealing a cycloid trajectory and providing explicit convergence rate estimates, thus deepening understanding of EM's dynamics.

Contribution

It introduces a novel trajectory-based analysis of EM in 2MLR, deriving explicit formulas and characterizing the cycloid path of iterations, and estimates the super-linear convergence exponent.

Findings

EM iterations follow a cycloid trajectory

Explicit closed-form EM update expressions across SNR regimes

Theoretical estimate for super-linear convergence exponent

Abstract

We study the trajectory of iterations and the convergence rates of the Expectation-Maximization (EM) algorithm for two-component Mixed Linear Regression (2MLR). The fundamental goal of MLR is to learn the regression models from unlabeled observations. The EM algorithm finds extensive applications in solving the mixture of linear regressions. Recent results have established the super-linear convergence of EM for 2MLR in the noiseless and high SNR settings under some assumptions and its global convergence rate with random initialization has been affirmed. However, the exponent of convergence has not been theoretically estimated and the geometric properties of the trajectory of EM iterations are not well-understood. In this paper, first, using Bessel functions we provide explicit closed-form expressions for the EM updates under all SNR regimes. Then, in the noiseless setting, we completely characterize the behavior of EM iterations by deriving a recurrence relation at the population level and notably show that all the iterations lie on a certain cycloid. Based on this new trajectory-based analysis, we exhibit the theoretical estimate for the exponent of super-linear convergence and further improve the statistical error bound at the finite-sample level. Our analysis provides a new framework for studying the behavior of EM for Mixed Linear Regression.