On Regularization and Inference with Label Constraints

TL;DR

This paper compares regularization with constraints and constrained inference in structured prediction, analyzing their effects on model performance, bias, and risk, and proposes combining both methods for improved results.

Contribution

It provides a theoretical comparison of two common label constraint strategies and introduces conditions for their effective combination to enhance model performance.

Findings

Regularization narrows the generalization gap but introduces bias.

Constrained inference reduces population risk by correcting violations.

Combining both approaches can mitigate biases and improve model outcomes.

Abstract

Prior knowledge and symbolic rules in machine learning are often expressed in the form of label constraints, especially in structured prediction problems. In this work, we compare two common strategies for encoding label constraints in a machine learning pipeline, regularization with constraints and constrained inference, by quantifying their impact on model performance. For regularization, we show that it narrows the generalization gap by precluding models that are inconsistent with the constraints. However, its preference for small violations introduces a bias toward a suboptimal model. For constrained inference, we show that it reduces the population risk by correcting a model's violation, and hence turns the violation into an advantage. Given these differences, we further explore the use of two approaches together and propose conditions for constrained inference to compensate for the bias introduced by regularization, aiming to improve both the model complexity and optimal risk.