Learning higher-order sequential structure with cloned HMMs

TL;DR

This paper introduces cloned HMMs (CHMMs), a biologically inspired sparse structure that enables efficient learning of variable order sequences, outperforming traditional models like n-grams, sequence memoizers, and LSTMs in language tasks.

Contribution

The paper proposes cloned HMMs with a simple sparsity constraint, allowing scalable training of large models that capture long-range temporal dependencies and uncertainty.

Findings

CHMMs outperform n-grams, sequence memoizers, and LSTMs in language modeling.

They can model long-distance dependencies and handle data with missing contexts.

Efficient training on GPUs enables models with over a billion parameters.

Abstract

Variable order sequence modeling is an important problem in artificial and natural intelligence. While overcomplete Hidden Markov Models (HMMs), in theory, have the capacity to represent long-term temporal structure, they often fail to learn and converge to local minima. We show that by constraining HMMs with a simple sparsity structure inspired by biology, we can make it learn variable order sequences efficiently. We call this model cloned HMM (CHMM) because the sparsity structure enforces that many hidden states map deterministically to the same emission state. CHMMs with over 1 billion parameters can be efficiently trained on GPUs without being severely affected by the credit diffusion problem of standard HMMs. Unlike n-grams and sequence memoizers, CHMMs can model temporal dependencies at arbitrarily long distances and recognize contexts with 'holes' in them. Compared to Recurrent Neural Networks and their Long Short-Term Memory extensions (LSTMs), CHMMs are generative models that can natively deal with uncertainty. Moreover, CHMMs return a higher-order graph that represents the temporal structure of the data which can be useful for community detection, and for building hierarchical models. Our experiments show that CHMMs can beat n-grams, sequence memoizers, and LSTMs on character-level language modeling tasks. CHMMs can be a viable alternative to these methods in some tasks that require variable order sequence modeling and the handling of uncertainty.