Agnostic Learning of Mixed Linear Regressions with EM and AM Algorithms

TL;DR

This paper demonstrates that EM and AM algorithms can effectively perform agnostic learning of mixed linear regressions, converging to optimal solutions without relying on specific generative assumptions, under certain conditions.

Contribution

It extends the analysis of EM and AM algorithms to agnostic settings, showing their convergence to population loss minimizers without assuming a specific data-generating model.

Findings

EM and AM algorithms converge to population loss minimizers in agnostic mixed linear regression.

Under standard conditions, algorithms achieve learning without realizable generative models.

Results demonstrate robustness of EM and AM in more general, less restrictive settings.

Abstract

Mixed linear regression is a well-studied problem in parametric statistics and machine learning. Given a set of samples, tuples of covariates and labels, the task of mixed linear regression is to find a small list of linear relationships that best fit the samples. Usually it is assumed that the label is generated stochastically by randomly selecting one of two or more linear functions, applying this chosen function to the covariates, and potentially introducing noise to the result. In that situation, the objective is to estimate the ground-truth linear functions up to some parameter error. The popular expectation maximization (EM) and alternating minimization (AM) algorithms have been previously analyzed for this. In this paper, we consider the more general problem of agnostic learning of mixed linear regression from samples, without such generative models. In particular, we show that the AM and EM algorithms, under standard conditions of separability and good initialization, lead to agnostic learning in mixed linear regression by converging to the population loss minimizers, for suitably defined loss functions. In some sense, this shows the strength of AM and EM algorithms that converges to ``optimal solutions'' even in the absence of realizable generative models.