Fast and scalable inference in hidden Markov models with Gaussian fields

TL;DR

This paper introduces a modified forward algorithm that enables fast, scalable inference in hidden Markov models incorporating Gaussian fields, effectively handling complex latent structures in high-dimensional time series data.

Contribution

It presents a novel inference method combining Gaussian fields with HMMs, leveraging sparsity in the Hessian to improve scalability and computational efficiency.

Findings

Efficient inference achieved in high-dimensional HMMs with Gaussian fields.

Demonstrated scalability through simulations and real case studies.

Enhanced detection and modeling capabilities in ecological and astrophysical data.

Abstract

Hidden Markov models (HMMs) are powerful tools for analysing time series data that depend on discrete underlying but unobserved states. As such, they have gained prominence across numerous empirical disciplines, in particular ecology, medicine, and economics. However, the increasing complexity of empirical data is often accompanied by additional latent structure such as spatial effects, temporal trends, or measurement perturbations. Gaussian fields provide an attractive building block for incorporating such structured latent variation into HMMs. Fast inference methods for Gaussian fields have emerged through the stochastic partial differential equation (SPDE) approach. Due to their sparse representation, these integrate well with novel frequentist estimation methods for random-effects models via the use of automatic differentiation and the Laplace approximation. Scaling to high dimensions requires tools such as (R)TMB to exploit sparsity in the Hessian w.r.t. the latent variables - a property satisfied by SPDE fields but violated by the HMM likelihood. We present a modified forward algorithm to compute the HMM likelihood, constructing sparsity in the Hessian and consequently enabling fast and scalable inference. We demonstrate the practical feasibility and the usefulness through simulations and two case studies exploring the detection of stellar flares as well as modelling the movement of lions.