Low-Rank Hidden State Embeddings for Viterbi Sequence Labeling

TL;DR

This paper introduces a novel latent state CRF model that learns embedded representations of output structure, enabling long-term dependency modeling and exact Viterbi inference, improving accuracy in sequence labeling tasks.

Contribution

It proposes a finite-state machine with factorized state transitions to embed long-term label dependencies, a significant advancement over traditional graphical models.

Findings

Achieves accuracy improvements in named entity recognition tasks.

Provides interpretable latent structures in complex sequence data.

Supports exact Viterbi inference with embedded long-term dependencies.

Abstract

In textual information extraction and other sequence labeling tasks it is now common to use recurrent neural networks (such as LSTM) to form rich embedded representations of long-term input co-occurrence patterns. Representation of output co-occurrence patterns is typically limited to a hand-designed graphical model, such as a linear-chain CRF representing short-term Markov dependencies among successive labels. This paper presents a method that learns embedded representations of latent output structure in sequence data. Our model takes the form of a finite-state machine with a large number of latent states per label (a latent variable CRF), where the state-transition matrix is factorized---effectively forming an embedded representation of state-transitions capable of enforcing long-term label dependencies, while supporting exact Viterbi inference over output labels. We demonstrate accuracy improvements and interpretable latent structure in a synthetic but complex task based on CoNLL named entity recognition.