Learning Constraints for Structured Prediction Using Rectifier Networks

TL;DR

This paper introduces a method to learn constraints for structured prediction in NLP tasks by training rectifier networks, which can improve accuracy especially with limited training data.

Contribution

It presents a novel approach to automatically learn output constraints using rectifier networks, reducing reliance on domain expertise.

Findings

Learned constraints improve prediction accuracy.

Method is effective with small training datasets.

Constraints can be converted into linear forms for inference.

Abstract

Various natural language processing tasks are structured prediction problems where outputs are constructed with multiple interdependent decisions. Past work has shown that domain knowledge, framed as constraints over the output space, can help improve predictive accuracy. However, designing good constraints often relies on domain expertise. In this paper, we study the problem of learning such constraints. We frame the problem as that of training a two-layer rectifier network to identify valid structures or substructures, and show a construction for converting a trained network into a system of linear constraints over the inference variables. Our experiments on several NLP tasks show that the learned constraints can improve the prediction accuracy, especially when the number of training examples is small.