DSDE: Using Proportion Estimation to Improve Model Selection for Out-of-Distribution Detection

TL;DR

This paper introduces DSDE, a novel method that leverages proportion estimation and change-point detection to improve out-of-distribution detection by better model selection and error rate estimation, significantly reducing false positives.

Contribution

The paper proposes a new approach called DSDE that inverts traditional p-value strategies and uses change-point detection for automatic hyperparameter tuning in OoD detection.

Findings

DSDE reduces FPR from 11.07% to 3.31% on CIFAR10.

Proposes a novel proportion estimation method with automatic hyperparameter selection.

Demonstrates improved OoD detection performance over single-model detectors.

Abstract

Model library is an effective tool for improving the performance of single-model Out-of-Distribution (OoD) detector, mainly through model selection and detector fusion. However, existing methods in the literature do not provide uncertainty quantification for model selection results. Additionally, the model ensemble process primarily focuses on controlling the True Positive Rate (TPR) while neglecting the False Positive Rate (FPR). In this paper, we emphasize the significance of the proportion of models in the library that identify the test sample as an OoD sample. This proportion holds crucial information and directly influences the error rate of OoD detection.To address this, we propose inverting the commonly-used sequential p-value strategies. We define the rejection region initially and then estimate the error rate. Furthermore, we introduce a novel perspective from change-point detection and propose an approach for proportion estimation with automatic hyperparameter selection. We name the proposed approach as DOS-Storey-based Detector Ensemble (DSDE). Experimental results on CIFAR10 and CIFAR100 demonstrate the effectiveness of our approach in tackling OoD detection challenges. Specifically, the CIFAR10 experiments show that DSDE reduces the FPR from 11.07% to 3.31% compared to the top-performing single-model detector.