Fitting large mixture models using stochastic component selection

TL;DR

This paper introduces a stochastic component selection method combining EM and Metropolis-Hastings to efficiently fit large mixture models, including deep and nonlinear variants, reducing computational costs.

Contribution

It presents a novel stochastic sampling approach for mixture model fitting that handles large component sets and complex transformations, improving efficiency and generality.

Findings

Effective in reducing computational costs for large mixture models

Applicable to deep and nonlinear mixture models

Demonstrated on synthetic and real datasets

Abstract

Traditional methods for unsupervised learning of finite mixture models require to evaluate the likelihood of all components of the mixture. This becomes computationally prohibitive when the number of components is large, as it is, for example, in the sum-product (transform) networks. Therefore, we propose to apply a combination of the expectation maximization and the Metropolis-Hastings algorithm to evaluate only a small number of, stochastically sampled, components, thus substantially reducing the computational cost. The Markov chain of component assignments is sequentially generated across the algorithm's iterations, having a non-stationary target distribution whose parameters vary via a gradient-descent scheme. We put emphasis on generality of our method, equipping it with the ability to train both shallow and deep mixture models which involve complex, and possibly nonlinear, transformations. The performance of our method is illustrated in a variety of synthetic and real-data contexts, considering deep models, such as mixtures of normalizing flows and sum-product (transform) networks.