Structured Prediction of Sequences and Trees using Infinite Contexts

TL;DR

This paper introduces a hierarchical, unbounded-context model for structured sequence and tree prediction, capturing global phenomena better than traditional Markov models, with promising empirical results in NLP tasks.

Contribution

The paper presents a novel hierarchical model with infinite contexts for structured prediction, utilizing a Pitman-Yor process prior for effective learning and smoothing.

Findings

Outperforms baseline finite-context Markov models in NLP tasks

Effective prediction algorithms based on A* and MCMC sampling

Demonstrates the importance of global context in structured prediction

Abstract

Linguistic structures exhibit a rich array of global phenomena, however commonly used Markov models are unable to adequately describe these phenomena due to their strong locality assumptions. We propose a novel hierarchical model for structured prediction over sequences and trees which exploits global context by conditioning each generation decision on an unbounded context of prior decisions. This builds on the success of Markov models but without imposing a fixed bound in order to better represent global phenomena. To facilitate learning of this large and unbounded model, we use a hierarchical Pitman-Yor process prior which provides a recursive form of smoothing. We propose prediction algorithms based on A* and Markov Chain Monte Carlo sampling. Empirical results demonstrate the potential of our model compared to baseline finite-context Markov models on part-of-speech tagging and syntactic parsing.