Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings

TL;DR

This study systematically evaluates various selective prediction methods across 17 NLP datasets in IID, OOD, and adversarial settings, revealing that most do not outperform the simple MaxProb baseline consistently and highlighting the need for cross-task evaluation.

Contribution

It provides a large-scale, comprehensive comparison of selective prediction approaches across multiple NLP tasks and settings, emphasizing the importance of cross-task evaluation for reliable assessment.

Findings

Most approaches do not consistently outperform MaxProb.

Performance varies significantly across tasks and settings.

Evaluation across multiple tasks is essential for reliable assessment.

Abstract

In order to equip NLP systems with selective prediction capability, several task-specific approaches have been proposed. However, which approaches work best across tasks or even if they consistently outperform the simplest baseline 'MaxProb' remains to be explored. To this end, we systematically study 'selective prediction' in a large-scale setup of 17 datasets across several NLP tasks. Through comprehensive experiments under in-domain (IID), out-of-domain (OOD), and adversarial (ADV) settings, we show that despite leveraging additional resources (held-out data/computation), none of the existing approaches consistently and considerably outperforms MaxProb in all three settings. Furthermore, their performance does not translate well across tasks. For instance, Monte-Carlo Dropout outperforms all other approaches on Duplicate Detection datasets but does not fare well on NLI datasets, especially in the OOD setting. Thus, we recommend that future selective prediction approaches should be evaluated across tasks and settings for reliable estimation of their capabilities.