Regularized joint mixture models

TL;DR

This paper introduces regularized joint mixture models that simultaneously learn latent group structures in heterogeneous data by coupling sparse graphical models for features and sparse regression models for responses, with theoretical convergence guarantees.

Contribution

It proposes a novel class of mixture models that jointly estimate feature and response distributions with regularization, capturing heterogeneity and confounding in paired data.

Findings

Effective in modeling heterogeneous data with latent groups

Joint estimation improves interpretability and confounding control

Theoretical convergence of the EM algorithm is established

Abstract

Regularized regression models are well studied and, under appropriate conditions, offer fast and statistically interpretable results. However, large data in many applications are heterogeneous in the sense of harboring distributional differences between latent groups. Then, the assumption that the conditional distribution of response Y given features X is the same for all samples may not hold. Furthermore, in scientific applications, the covariance structure of the features may contain important signals and its learning is also affected by latent group structure. We propose a class of mixture models for paired data (X, Y) that couples together the distribution of X (using sparse graphical models) and the conditional Y | X (using sparse regression models). The regression and graphical models are specific to the latent groups and model parameters are estimated jointly (hence the name "regularized joint mixtures"). This allows signals in either or both of the feature distribution and regression model to inform learning of latent structure and provides automatic control of confounding by such structure. Estimation is handled via an expectation-maximization algorithm, whose convergence is established theoretically. We illustrate the key ideas via empirical examples. An R package is available at https://github.com/k-perrakis/regjmix.