Penalized estimation in high-dimensional hidden Markov models with state-specific graphical models

TL;DR

This paper introduces a penalized estimation method for high-dimensional hidden Markov models with multivariate normal observations, enabling sparse, interpretable state-specific graphs and automatic adaptation to data complexity.

Contribution

It proposes a novel adaptive $ ext{L}_1$-penalization technique for inverse covariance matrices in HMMs, handling unknown states and scaling issues without manual tuning.

Findings

Effective in simulated data scenarios

Improves predictive power in genome biology data

Produces richer, interpretable state-specific graphs

Abstract

We consider penalized estimation in hidden Markov models (HMMs) with multivariate Normal observations. In the moderate-to-large dimensional setting, estimation for HMMs remains challenging in practice, due to several concerns arising from the hidden nature of the states. We address these concerns by $\ell_1$-penalization of state-specific inverse covariance matrices. Penalized estimation leads to sparse inverse covariance matrices which can be interpreted as state-specific conditional independence graphs. Penalization is nontrivial in this latent variable setting; we propose a penalty that automatically adapts to the number of states $K$ and the state-specific sample sizes and can cope with scaling issues arising from the unknown states. The methodology is adaptive and very general, applying in particular to both low- and high-dimensional settings without requiring hand tuning. Furthermore, our approach facilitates exploration of the number of states $K$ by coupling estimation for successive candidate values $K$. Empirical results on simulated examples demonstrate the effectiveness of the proposed approach. In a challenging real data example from genome biology, we demonstrate the ability of our approach to yield gains in predictive power and to deliver richer estimates than existing methods.