Tuning Out-of-Distribution (OOD) Detectors Without Given OOD Data

TL;DR

This paper addresses the challenge of tuning out-of-distribution (OOD) detectors without relying on any OOD data, proposing a new approach that improves detection performance across various detector families.

Contribution

It introduces the formal problem of tuning OOD detectors without OOD data and proposes a generic method that enhances performance without extra data.

Findings

The proposed method outperforms baseline approaches across high-parameter OOD detectors.

Performance is comparable to existing methods for lower-parameter detectors.

Variance in detector performance depends significantly on the choice of adhoc datasets.

Abstract

Existing out-of-distribution (OOD) detectors are often tuned by a separate dataset deemed OOD with respect to the training distribution of a neural network (NN). OOD detectors process the activations of NN layers and score the output, where parameters of the detectors are determined by fitting to an in-distribution (training) set and the aforementioned dataset chosen adhocly. At detector training time, this adhoc dataset may not be available or difficult to obtain, and even when it's available, it may not be representative of actual OOD data, which is often ''unknown unknowns." Current benchmarks may specify some left-out set from test OOD sets. We show that there can be significant variance in performance of detectors based on the adhoc dataset chosen in current literature, and thus even if such a dataset can be collected, the performance of the detector may be highly dependent on the choice. In this paper, we introduce and formalize the often neglected problem of tuning OOD detectors without a given ``OOD'' dataset. To this end, we present strong baselines as an attempt to approach this problem. Furthermore, we propose a new generic approach to OOD detector tuning that does not require any extra data other than those used to train the NN. We show that our approach improves over baseline methods consistently across higher-parameter OOD detector families, while being comparable across lower-parameter families.