Reified Context Models

TL;DR

Reified context models dynamically select context size during inference, balancing expressivity and coverage, which improves natural language processing tasks.

Contribution

This paper introduces reified context models that treat context size as a variable within the model, enabling adaptive inference.

Findings

Achieves a balance between expressivity and coverage.

Improves performance on three natural language tasks.

Demonstrates the effectiveness of reifying context size.

Abstract

A classic tension exists between exact inference in a simple model and approximate inference in a complex model. The latter offers expressivity and thus accuracy, but the former provides coverage of the space, an important property for confidence estimation and learning with indirect supervision. In this work, we introduce a new approach, reified context models, to reconcile this tension. Specifically, we let the amount of context (the arity of the factors in a graphical model) be chosen "at run-time" by reifying it---that is, letting this choice itself be a random variable inside the model. Empirically, we show that our approach obtains expressivity and coverage on three natural language tasks.