Variational approximation for mixtures of linear mixed models

TL;DR

This paper introduces a variational approach for fitting mixtures of linear mixed models, enabling faster parameter estimation and automatic model selection, with improved efficiency through reparametrization techniques.

Contribution

It develops a variational greedy algorithm for simultaneous parameter estimation and model selection in MLMMs, improving speed and automation over traditional methods.

Findings

Variational methods provide closed-form updates for MLMMs.

The proposed algorithm automatically determines the number of mixture components.

Reparametrization via hierarchical centering improves convergence efficiency.

Abstract

Mixtures of linear mixed models (MLMMs) are useful for clustering grouped data and can be estimated by likelihood maximization through the EM algorithm. The conventional approach to determining a suitable number of components is to compare different mixture models using penalized log-likelihood criteria such as BIC.We propose fitting MLMMs with variational methods which can perform parameter estimation and model selection simultaneously. A variational approximation is described where the variational lower bound and parameter updates are in closed form, allowing fast evaluation. A new variational greedy algorithm is developed for model selection and learning of the mixture components. This approach allows an automatic initialization of the algorithm and returns a plausible number of mixture components automatically. In cases of weak identifiability of certain model parameters, we use hierarchical centering to reparametrize the model and show empirically that there is a gain in efficiency by variational algorithms similar to that in MCMC algorithms. Related to this, we prove that the approximate rate of convergence of variational algorithms by Gaussian approximation is equal to that of the corresponding Gibbs sampler which suggests that reparametrizations can lead to improved convergence in variational algorithms as well.