Mitigating the Modality Gap: Few-Shot Out-of-Distribution Detection with Multi-modal Prototypes and Image Bias Estimation

TL;DR

This paper introduces a novel framework called SUPREME that improves vision-language model-based out-of-distribution detection by incorporating image prototypes, bias estimation, and a new OOD scoring method, significantly reducing false positives.

Contribution

The paper proposes a new few-shot tuning framework, SUPREME, that enhances OOD detection by reducing the modality gap through biased prompts and image-text consistency, without additional training.

Findings

SUPREME outperforms existing VLM-based OOD detection methods.

Incorporating image prototypes reduces false positives.

The new OOD score $S_{GMP}$ improves detection accuracy.

Abstract

Existing vision-language model (VLM)-based methods for out-of-distribution (OOD) detection typically rely on similarity scores between input images and in-distribution (ID) text prototypes. However, the modality gap between image and text often results in high false positive rates, as OOD samples can exhibit high similarity to ID text prototypes. To mitigate the impact of this modality gap, we propose incorporating ID image prototypes along with ID text prototypes. We present theoretical analysis and empirical evidence indicating that this approach enhances VLM-based OOD detection performance without any additional training. To further reduce the gap between image and text, we introduce a novel few-shot tuning framework, SUPREME, comprising biased prompts generation (BPG) and image-text consistency (ITC) modules. BPG enhances image-text fusion and improves generalization by conditioning ID text prototypes on the Gaussian-based estimated image domain bias; ITC reduces the modality gap by minimizing intra- and inter-modal distances. Moreover, inspired by our theoretical and empirical findings, we introduce a novel OOD score $S_{\textit{GMP}}$, leveraging uni- and cross-modal similarities. Finally, we present extensive experiments to demonstrate that SUPREME consistently outperforms existing VLM-based OOD detection methods.