Combining Constrained and Unconstrained Decoding via Boosting: BoostCD and Its Application to Information Extraction

TL;DR

This paper introduces BoostCD, a novel method that combines constrained and unconstrained decoding to improve structured NLP tasks, demonstrated through a new approach called BoostIE for information extraction.

Contribution

The paper proposes BoostCD, a two-phase boosting framework that leverages both constrained and unconstrained decoding, enhancing output quality without retraining the model.

Findings

BoostIE outperforms prior methods in information extraction tasks.

BoostCD effectively exploits complementary errors of constrained and unconstrained decoding.

BoostIE maintains performance both in-distribution and out-of-distribution.

Abstract

Many recent approaches to structured NLP tasks use an autoregressive language model $M$ to map unstructured input text $x$ to output text $y$ representing structured objects (such as tuples, lists, trees, code, etc.), where the desired output structure is enforced via constrained decoding. During training, these approaches do not require the model to be aware of the constraints, which are merely implicit in the training outputs $y$. This is advantageous as it allows for dynamic constraints without requiring retraining, but can lead to low-quality output during constrained decoding at test time. We overcome this problem with Boosted Constrained Decoding (BoostCD), which combines constrained and unconstrained decoding in two phases: Phase 1 decodes from the base model $M$ twice, in constrained and unconstrained mode, obtaining two weak predictions. In phase 2, a learned autoregressive boosted model combines the two weak predictions into one final prediction. The mistakes made by the base model with vs. without constraints tend to be complementary, which the boosted model learns to exploit for improved performance. We demonstrate the power of BoostCD by applying it to closed information extraction. Our model, BoostIE, outperforms prior approaches both in and out of distribution, addressing several common errors identified in those approaches.