Learning to Reject with a Fixed Predictor: Application to Decontextualization

TL;DR

This paper introduces a new approach for classification with a reject option, specifically applied to decontextualization in NLP, with theoretical guarantees and improved performance on a new dataset.

Contribution

It proposes a novel problem formulation and surrogate loss function for classification with reject options, along with a theoretical analysis and application to decontextualization.

Findings

Significant 25% coverage improvement over baselines

Error rate halved while maintaining high coverage

Approaching the theoretical performance limit

Abstract

We study the problem of classification with a reject option for a fixed predictor, applicable in natural language processing. We introduce a new problem formulation for this scenario, and an algorithm minimizing a new surrogate loss function. We provide a complete theoretical analysis of the surrogate loss function with a strong $H$-consistency guarantee. For evaluation, we choose the decontextualization task, and provide a manually-labelled dataset of $2\mathord,000$ examples. Our algorithm significantly outperforms the baselines considered, with a $\sim\!\!25\%$ improvement in coverage when halving the error rate, which is only $\sim\!\! 3 \%$ away from the theoretical limit.