Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP

TL;DR

This paper introduces ZOC, a novel zero-shot method leveraging CLIP and a text-based image description generator to effectively detect out-of-distribution samples without training on OOD data.

Contribution

It extends CLIP with a text-based image description generator and uses it for zero-shot OOD detection, outperforming existing baselines on benchmark datasets.

Findings

ZOC significantly outperforms baseline methods in OOD detection accuracy.

The method effectively generates candidate unknown class names for test samples.

ZOC demonstrates strong zero-shot OOD detection capabilities across multiple datasets.

Abstract

In an out-of-distribution (OOD) detection problem, samples of known classes(also called in-distribution classes) are used to train a special classifier. In testing, the classifier can (1) classify the test samples of known classes to their respective classes and also (2) detect samples that do not belong to any of the known classes (i.e., they belong to some unknown or OOD classes). This paper studies the problem of zero-shot out-of-distribution(OOD) detection, which still performs the same two tasks in testing but has no training except using the given known class names. This paper proposes a novel yet simple method (called ZOC) to solve the problem. ZOC builds on top of the recent advances in zero-shot classification through multi-modal representation learning. It first extends the pre-trained language-vision model CLIP by training a text-based image description generator on top of CLIP. In testing, it uses the extended model to generate candidate unknown class names for each test sample and computes a confidence score based on both the known class names and candidate unknown class names for zero-shot OOD detection. Experimental results on 5 benchmark datasets for OOD detection demonstrate that ZOC outperforms the baselines by a large margin.