Estimating the Number of Components in Finite Mixture Models via Variational Approximation

TL;DR

This paper proposes a variational Bayes-based method for selecting the number of components in finite mixture models, demonstrating theoretical consistency and stable behavior under model misspecification.

Contribution

It introduces a new ELBO-based model selection approach that does not rely on conjugate priors and proves its consistency and stability theoretically.

Findings

ELBO maximization leads to consistent component number selection.

MF approximation inherits posterior stability, reducing overfitting.

Empirical results validate theoretical claims and outperform existing methods.

Abstract

This work introduces a new method for selecting the number of components in finite mixture models (FMMs) using variational Bayes, inspired by the large-sample properties of the Evidence Lower Bound (ELBO) derived from mean-field (MF) variational approximation. Specifically, we establish matching upper and lower bounds for the ELBO without assuming conjugate priors, suggesting the consistency of model selection for FMMs based on maximizing the ELBO. As a by-product of our proof, we demonstrate that the MF approximation inherits the stable behavior (benefited from model singularity) of the posterior distribution, which tends to eliminate the extra components under model misspecification where the number of mixture components is over-specified. This stable behavior also leads to the $n^{-1/2}$ convergence rate for parameter estimation, up to a logarithmic factor, under this model overspecification. Empirical experiments are conducted to validate our theoretical findings and compare with other state-of-the-art methods for selecting the number of components in FMMs.