Switching Autoregressive Low-rank Tensor Models

TL;DR

This paper introduces SALT models that combine the strengths of ARHMMs and SLDSs for time-series analysis, enabling efficient long-range dependency modeling with interpretability and reduced overfitting.

Contribution

The paper proposes SALT models with low-rank tensor parameterization, bridging ARHMMs and SLDSs, and provides theoretical analysis and empirical validation.

Findings

SALT models outperform traditional ARHMMs and SLDSs on prediction tasks.

Low-rank tensor factorization reduces overfitting and captures long-range dependencies.

Learned tensors offer insights into temporal dynamics within states.

Abstract

An important problem in time-series analysis is modeling systems with time-varying dynamics. Probabilistic models with joint continuous and discrete latent states offer interpretable, efficient, and experimentally useful descriptions of such data. Commonly used models include autoregressive hidden Markov models (ARHMMs) and switching linear dynamical systems (SLDSs), each with its own advantages and disadvantages. ARHMMs permit exact inference and easy parameter estimation, but are parameter intensive when modeling long dependencies, and hence are prone to overfitting. In contrast, SLDSs can capture long-range dependencies in a parameter efficient way through Markovian latent dynamics, but present an intractable likelihood and a challenging parameter estimation task. In this paper, we propose switching autoregressive low-rank tensor (SALT) models, which retain the advantages of both approaches while ameliorating the weaknesses. SALT parameterizes the tensor of an ARHMM with a low-rank factorization to control the number of parameters and allow longer range dependencies without overfitting. We prove theoretical and discuss practical connections between SALT, linear dynamical systems, and SLDSs. We empirically demonstrate quantitative advantages of SALT models on a range of simulated and real prediction tasks, including behavioral and neural datasets. Furthermore, the learned low-rank tensor provides novel insights into temporal dependencies within each discrete state.