Improving Out-of-Distribution Detection by Combining Existing Post-hoc Methods

TL;DR

This paper explores combining multiple existing post-hoc OOD detection methods to improve robustness, proposing strategies for integration, extending evaluation metrics, and providing guidelines for practical use without known OOD data.

Contribution

It introduces four novel strategies for combining OOD detection scores, extending evaluation metrics for multi-dimensional detectors, and offers practical guidelines for real-world scenarios.

Findings

Combined methods outperform individual detectors on benchmarks.

New evaluation metrics enable comprehensive assessment of multi-detectors.

Guidelines help select effective combinations without prior OOD data.

Abstract

Since the seminal paper of Hendrycks et al. arXiv:1610.02136, Post-hoc deep Out-of-Distribution (OOD) detection has expanded rapidly. As a result, practitioners working on safety-critical applications and seeking to improve the robustness of a neural network now have a plethora of methods to choose from. However, no method outperforms every other on every dataset arXiv:2210.07242, so the current best practice is to test all the methods on the datasets at hand. This paper shifts focus from developing new methods to effectively combining existing ones to enhance OOD detection. We propose and compare four different strategies for integrating multiple detection scores into a unified OOD detector, based on techniques such as majority vote, empirical and copulas-based Cumulative Distribution Function modeling, and multivariate quantiles based on optimal transport. We extend common OOD evaluation metrics -- like AUROC and FPR at fixed TPR rates -- to these multi-dimensional OOD detectors, allowing us to evaluate them and compare them with individual methods on extensive benchmarks. Furthermore, we propose a series of guidelines to choose what OOD detectors to combine in more realistic settings, i.e. in the absence of known OOD data, relying on principles drawn from Outlier Exposure arXiv:1812.04606. The code is available at https://github.com/paulnovello/multi-ood.