Exploring the likelihood surface in multivariate Gaussian mixtures using Hamiltonian Monte Carlo

TL;DR

This paper introduces a method combining Hamiltonian Monte Carlo with local optimization to explore the likelihood surface of multivariate Gaussian mixtures, aiming to identify global maxima and assess multimodality.

Contribution

The authors propose a novel approach that integrates HMC sampling with local optimization to better explore the likelihood landscape in Gaussian mixture models.

Findings

Successfully identified the global maximum likelihood estimate for the tested dataset.

Demonstrated the potential of HMC to explore complex likelihood surfaces in mixture models.

Faced challenges with high-dimensional models, prompting the use of various tricks to improve mixing.

Abstract

Multimodality of the likelihood in Gaussian mixtures is a well-known problem. The choice of the initial parameter vector for the numerical optimizer may affect whether the optimizer finds the global maximum, or gets trapped in a local maximum of the likelihood. We propose to use Hamiltonian Monte Carlo (HMC) to explore the part of the parameter space which has a high likelihood. Each sampled parameter vector is used as the initial value for quasi-Newton optimizer, and the resulting sample of (maximum) likelihood values is used to determine if the likelihood is multimodal. We use a single simulated data set from a three component bivariate mixture to develop and test the method. We use state-of-the-art HCM software, but experience difficulties when trying to directly apply HMC to the full model with 15 parameters. To improve the mixing of the Markov Chain we explore various tricks, and conclude that for the dataset at hand we have found the global maximum likelihood estimate.