A Nonparametric Maximum Likelihood Approach to Mixture of Regression

TL;DR

This paper introduces a fully nonparametric maximum likelihood estimator for mixture of linear regression models, providing theoretical guarantees and demonstrating practical effectiveness in estimating conditional densities and mixing distributions.

Contribution

It develops a novel nonparametric maximum likelihood approach for mixture of regression models, with theoretical guarantees and practical algorithms, surpassing parametric methods.

Findings

Estimator exists under broad conditions.

Achieves near-parametric rates in estimating conditional density.

Performs well in numerical experiments and enables individualized inference.

Abstract

We study mixture of linear regression (random coefficient) models, which capture population heterogeneity by allowing the regression coefficients to follow an unknown distribution $G^*$. In contrast to common parametric methods that fix the mixing distribution form and rely on the EM algorithm, we develop a fully nonparametric maximum likelihood estimator (NPMLE). We show that this estimator exists under broad conditions and can be computed via a discrete approximation procedure inspired by the exemplar method. We further establish theoretical guarantees demonstrating that the NPMLE achieves near-parametric rates in estimating the conditional density of $Y|X$, both for fixed and random designs, when $\sigma$ is known and $G^*$ has compact support. In the random design setting, we also prove consistency of the estimated mixing distribution in the L\'evy-Prokhorov distance. Numerical experiments indicate that our approach performs well and additionally enables posterior-based individualized coefficient inference through an empirical Bayes framework.