DOI: Divergence-based Out-of-Distribution Indicators via Deep Generative Models

TL;DR

This paper introduces a new divergence-based framework for out-of-distribution detection in deep generative models, benchmarks existing methods extensively, and proposes a simple fine-tuning algorithm that significantly improves OoD detection performance.

Contribution

It presents the first large-scale benchmark for OoD indicators, proposes a novel divergence-based framework, and introduces a simple fine-tuning method that outperforms previous approaches.

Findings

Existing OoD indicators perform poorly on large benchmarks.

The proposed Single-shot Fine-tune method improves AUROC by 5-8 points.

Likelihood-based OoD detection is ineffective, and the new criterion is more reliable.

Abstract

To ensure robust and reliable classification results, OoD (out-of-distribution) indicators based on deep generative models are proposed recently and are shown to work well on small datasets. In this paper, we conduct the first large collection of benchmarks (containing 92 dataset pairs, which is 1 order of magnitude larger than previous ones) for existing OoD indicators and observe that none perform well. We thus advocate that a large collection of benchmarks is mandatory for evaluating OoD indicators. We propose a novel theoretical framework, DOI, for divergence-based Out-of-Distribution indicators (instead of traditional likelihood-based) in deep generative models. Following this framework, we further propose a simple and effective OoD detection algorithm: Single-shot Fine-tune. It significantly outperforms past works by 5~8 in AUROC, and its performance is close to optimal. In recent, the likelihood criterion is shown to be ineffective in detecting OoD. Single-shot Fine-tune proposes a novel fine-tune criterion to detect OoD, by whether the likelihood of the testing sample is improved after fine-tuning a well-trained model on it. Fine-tune criterion is a clear and easy-following criterion, which will lead the OoD domain into a new stage.