Reduction of Maximum Entropy Models to Hidden Markov Models

TL;DR

This paper demonstrates that maximum entropy models can be represented as hidden Markov models, enabling new modeling capabilities and training methods, with experimental validation showing improved performance on a word disambiguation task.

Contribution

It introduces a method to model maxent models using HMMs, allowing hidden variables and sequences, and provides a training approach via the forward-backward algorithm.

Findings

Maxent models can be represented as HMMs.

The approach enables training with hidden variables.

Experimental results show improved disambiguation performance.

Abstract

We show that maximum entropy (maxent) models can be modeled with certain kinds of HMMs, allowing us to construct maxent models with hidden variables, hidden state sequences, or other characteristics. The models can be trained using the forward-backward algorithm. While the results are primarily of theoretical interest, unifying apparently unrelated concepts, we also give experimental results for a maxent model with a hidden variable on a word disambiguation task; the model outperforms standard techniques.