Delving into Out-of-Distribution Detection with Vision-Language Representations

TL;DR

This paper introduces Maximum Concept Matching (MCM), a multi-modal, zero-shot out-of-distribution detection method leveraging vision-language representations, demonstrating significant performance improvements over single-modal approaches.

Contribution

It proposes MCM, a novel multi-modal OOD detection technique that aligns visual features with textual concepts, supported by theoretical analysis and extensive empirical validation.

Findings

MCM outperforms visual-only baselines by 13.1% AUROC on challenging OOD tasks.

The method effectively leverages vision-language pre-training for improved OOD detection.

Theoretical insights explain the success of multi-modal alignment in OOD detection.

Abstract

Recognizing out-of-distribution (OOD) samples is critical for machine learning systems deployed in the open world. The vast majority of OOD detection methods are driven by a single modality (e.g., either vision or language), leaving the rich information in multi-modal representations untapped. Inspired by the recent success of vision-language pre-training, this paper enriches the landscape of OOD detection from a single-modal to a multi-modal regime. Particularly, we propose Maximum Concept Matching (MCM), a simple yet effective zero-shot OOD detection method based on aligning visual features with textual concepts. We contribute in-depth analysis and theoretical insights to understand the effectiveness of MCM. Extensive experiments demonstrate that MCM achieves superior performance on a wide variety of real-world tasks. MCM with vision-language features outperforms a common baseline with pure visual features on a hard OOD task with semantically similar classes by 13.1% (AUROC). Code is available at https://github.com/deeplearning-wisc/MCM.