Sharp analysis of EM for learning mixtures of pairwise differences

TL;DR

This paper provides a detailed analysis of the EM algorithm's convergence properties for learning mixtures of pairwise differences, revealing conditions for linear convergence and optimal estimation rates.

Contribution

It offers the first local convergence analysis of EM for this problem, establishing sharp estimation bounds and highlighting the impact of covariate structure on convergence.

Findings

EM converges linearly near the ground truth.

The limit of EM achieves the optimal estimation rate.

Random initialization may not guarantee convergence.

Abstract

We consider a symmetric mixture of linear regressions with random samples from the pairwise comparison design, which can be seen as a noisy version of a type of Euclidean distance geometry problem. We analyze the expectation-maximization (EM) algorithm locally around the ground truth and establish that the sequence converges linearly, providing an $\ell_\infty$-norm guarantee on the estimation error of the iterates. Furthermore, we show that the limit of the EM sequence achieves the sharp rate of estimation in the $\ell_2$-norm, matching the information-theoretically optimal constant. We also argue through simulation that convergence from a random initialization is much more delicate in this setting, and does not appear to occur in general. Our results show that the EM algorithm can exhibit several unique behaviors when the covariate distribution is suitably structured.