An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks

TL;DR

This paper introduces high-order energy terms with neural parameterizations to improve sequence labeling by capturing complex label dependencies, demonstrating significant empirical gains across multiple NLP tasks without sacrificing decoding speed.

Contribution

It proposes a novel framework using neural energy-based inference networks to model high-order label dependencies in sequence labeling tasks.

Findings

Substantial performance improvements on four sequence labeling tasks.

High-order energies enhance robustness in noisy data conditions.

Same decoding speed as local classifiers.

Abstract

Many tasks in natural language processing involve predicting structured outputs, e.g., sequence labeling, semantic role labeling, parsing, and machine translation. Researchers are increasingly applying deep representation learning to these problems, but the structured component of these approaches is usually quite simplistic. In this work, we propose several high-order energy terms to capture complex dependencies among labels in sequence labeling, including several that consider the entire label sequence. We use neural parameterizations for these energy terms, drawing from convolutional, recurrent, and self-attention networks. We use the framework of learning energy-based inference networks (Tu and Gimpel, 2018) for dealing with the difficulties of training and inference with such models. We empirically demonstrate that this approach achieves substantial improvement using a variety of high-order energy terms on four sequence labeling tasks, while having the same decoding speed as simple, local classifiers. We also find high-order energies to help in noisy data conditions.