Imbalanced Mixed Linear Regression

TL;DR

This paper introduces Mix-IRLS, a new algorithm for mixed linear regression that performs well in imbalanced, small sample, and outlier-rich scenarios, outperforming existing methods.

Contribution

The paper proposes Mix-IRLS, a simple, fast, and effective sequential approach for mixed linear regression, especially in imbalanced and challenging data conditions.

Findings

Mix-IRLS outperforms existing methods in imbalanced mixtures.

It is effective with small sample sizes and outliers.

Provides theoretical recovery guarantees for imbalanced cases.

Abstract

We consider the problem of mixed linear regression (MLR), where each observed sample belongs to one of $K$ unknown linear models. In practical applications, the proportions of the $K$ components are often imbalanced. Unfortunately, most MLR methods do not perform well in such settings. Motivated by this practical challenge, in this work we propose Mix-IRLS, a novel, simple and fast algorithm for MLR with excellent performance on both balanced and imbalanced mixtures. In contrast to popular approaches that recover the $K$ models simultaneously, Mix-IRLS does it sequentially using tools from robust regression. Empirically, Mix-IRLS succeeds in a broad range of settings where other methods fail. These include imbalanced mixtures, small sample sizes, presence of outliers, and an unknown number of models $K$. In addition, Mix-IRLS outperforms competing methods on several real-world datasets, in some cases by a large margin. We complement our empirical results by deriving a recovery guarantee for Mix-IRLS, which highlights its advantage on imbalanced mixtures.