Strong identifiability and parameter learning in regression with heterogeneous response

TL;DR

This paper studies the theoretical properties of mixture of regression models, including identifiability, convergence rates, and Bayesian posterior behavior, with practical simulations and data examples.

Contribution

It provides new conditions for strong identifiability and analyzes convergence and Bayesian contraction in mixture regression models, including over-fitted and unknown component scenarios.

Findings

Conditions for strong identifiability established

Convergence rates for parameter estimation derived

Bayesian posterior contraction analyzed

Abstract

Mixtures of regression are a powerful class of models for regression learning with respect to a highly uncertain and heterogeneous response variable of interest. In addition to being a rich predictive model for the response given some covariates, the parameters in this model class provide useful information about the heterogeneity in the data population, which is represented by the conditional distributions for the response given the covariates associated with a number of distinct but latent subpopulations. In this paper, we investigate conditions of strong identifiability, rates of convergence for conditional density and parameter estimation, and the Bayesian posterior contraction behavior arising in finite mixture of regression models, under exact-fitted and over-fitted settings and when the number of components is unknown. This theory is applicable to common choices of link functions and families of conditional distributions employed by practitioners. We provide simulation studies and data illustrations, which shed some light on the parameter learning behavior found in several popular regression mixture models reported in the literature.