Deep Neural Network Benchmarks for Selective Classification

TL;DR

This paper benchmarks 18 selective classification methods based on deep neural networks across 44 diverse datasets, providing comprehensive insights into their relative performance and suitability for different objectives in trustworthy AI deployment.

Contribution

It offers the first extensive empirical comparison of multiple deep learning-based selective classification approaches on a large, diverse dataset collection.

Findings

No single method outperforms others across all metrics.

Performance varies depending on the specific evaluation criterion and dataset.

Practitioners should choose methods based on their specific accuracy and coverage needs.

Abstract

With the increasing deployment of machine learning models in many socially sensitive tasks, there is a growing demand for reliable and trustworthy predictions. One way to accomplish these requirements is to allow a model to abstain from making a prediction when there is a high risk of making an error. This requires adding a selection mechanism to the model, which selects those examples for which the model will provide a prediction. The selective classification framework aims to design a mechanism that balances the fraction of rejected predictions (i.e., the proportion of examples for which the model does not make a prediction) versus the improvement in predictive performance on the selected predictions. Multiple selective classification frameworks exist, most of which rely on deep neural network architectures. However, the empirical evaluation of the existing approaches is still limited to partial comparisons among methods and settings, providing practitioners with little insight into their relative merits. We fill this gap by benchmarking 18 baselines on a diverse set of 44 datasets that includes both image and tabular data. Moreover, there is a mix of binary and multiclass tasks. We evaluate these approaches using several criteria, including selective error rate, empirical coverage, distribution of rejected instance's classes, and performance on out-of-distribution instances. The results indicate that there is not a single clear winner among the surveyed baselines, and the best method depends on the users' objectives.