Joint Training for Selective Prediction

TL;DR

This paper introduces a joint-training method for selective prediction in NLP classifiers, optimizing both the classifier and deferral policy simultaneously to improve trustworthiness and performance in human-in-the-loop systems.

Contribution

It proposes a novel joint-training approach that enhances selective prediction by simultaneously optimizing classifier representations and deferral policies, outperforming existing baselines.

Findings

Joint training improves selective prediction outcomes.

Method enhances performance of both classifier and deferral modules.

Results are validated on four classification tasks.

Abstract

Classifier models are prevalent in natural language processing (NLP), often with high accuracy. Yet in real world settings, human-in-the-loop systems can foster trust in model outputs and even higher performance. Selective Prediction (SP) methods determine when to adopt a classifier's output versus defer to a human. Previous SP approaches have addressed how to improve softmax as a measure of model confidence, or have developed separate confidence estimators. One previous method involves learning a deferral model based on engineered features. We introduce a novel joint-training approach that simultaneously optimizes learned representations used by the classifier module and a learned deferral policy. Our results on four classification tasks demonstrate that joint training not only leads to better SP outcomes over two strong baselines, but also improves the performance of both modules.