Boosting Stochastic Optimisation for High-dimensional Latent Variable Models

TL;DR

This paper proposes enhanced stochastic optimisation methods, combining Langevin sampling and minibatch gradients, to efficiently estimate high-dimensional latent variable models, demonstrated through simulations and a personality test application.

Contribution

It introduces a novel combination of Langevin sampling and minibatch techniques to improve stochastic optimisation for high-dimensional latent variable models.

Findings

Combined strategies outperform competitors in simulations.

Method scales effectively to 30 latent dimensions.

Efficient sampling reduces computational complexity.

Abstract

Latent variable models are widely used in social and behavioural sciences, including education, psychology, and political science. With the increasing availability of large and complex datasets, high-dimensional latent variable models have become more common. However, estimating such models via marginal maximum likelihood is computationally challenging because it requires evaluating a large number of high-dimensional integrals. Stochastic optimisation, which combines stochastic approximation and sampling techniques, has been shown to be effective. It iterates between sampling latent variables from their posterior distribution under current parameter estimates and updating the model parameters using an approximate stochastic gradient constructed from the latent variable samples. In this paper, we investigate strategies to improve the performance of stochastic optimisation for high-dimensional latent variable models. The improvement is achieved through two strategies: a Metropolis-adjusted Langevin sampler that uses the gradient of the negative complete-data log-likelihood to sample latent variables efficiently, and a minibatch gradient technique that uses only a subset of observations when sampling latent variables and constructing stochastic gradients. Our simulation studies show that combining these strategies yields the best overall performance among competitors. An application to a personality test with 30 latent dimensions further demonstrates that the proposed algorithm scales effectively to high-dimensional settings.