Reject option models comprising out-of-distribution detection

TL;DR

This paper introduces new reject option models for out-of-distribution detection, defining optimal strategies, proposing double-score methods, and developing novel evaluation metrics to improve OOD detection performance.

Contribution

It presents three new OOD reject models, identifies common optimal strategies, and introduces double-score detection methods and new evaluation metrics.

Findings

Double-score methods outperform state-of-the-art OOD detection techniques.

Proposed evaluation metrics provide more reliable assessment of OOD methods.

All models share a common class of optimal strategies despite different formulations.

Abstract

The optimal prediction strategy for out-of-distribution (OOD) setups is a fundamental question in machine learning. In this paper, we address this question and present several contributions. We propose three reject option models for OOD setups: the Cost-based model, the Bounded TPR-FPR model, and the Bounded Precision-Recall model. These models extend the standard reject option models used in non-OOD setups and define the notion of an optimal OOD selective classifier. We establish that all the proposed models, despite their different formulations, share a common class of optimal strategies. Motivated by the optimal strategy, we introduce double-score OOD methods that leverage uncertainty scores from two chosen OOD detectors: one focused on OOD/ID discrimination and the other on misclassification detection. The experimental results consistently demonstrate the superior performance of this simple strategy compared to state-of-the-art methods. Additionally, we propose novel evaluation metrics derived from the definition of the optimal strategy under the proposed OOD rejection models. These new metrics provide a comprehensive and reliable assessment of OOD methods without the deficiencies observed in existing evaluation approaches.