Out-of-Distribution Detection with Positive and Negative Prompt Supervision Using Large Language Models

TL;DR

This paper introduces a novel positive and negative prompt supervision method using large language models to improve out-of-distribution detection in vision-language models, achieving superior results on standard benchmarks.

Contribution

It proposes a new prompt supervision approach that leverages class-specific prompts and a graph-based architecture to enhance OOD detection performance.

Findings

Outperforms state-of-the-art methods on CIFAR-100 and ImageNet-1K benchmarks.

Effective across multiple OOD datasets and LLM configurations.

Enhances energy-based OOD detection with semantic-aware supervision.

Abstract

Out-of-distribution (OOD) detection is committed to delineating the classification boundaries between in-distribution (ID) and OOD images. Recent advances in vision-language models (VLMs) have demonstrated remarkable OOD detection performance by integrating both visual and textual modalities. In this context, negative prompts are introduced to emphasize the dissimilarity between image features and prompt content. However, these prompts often include a broad range of non-ID features, which may result in suboptimal outcomes due to the capture of overlapping or misleading information. To address this issue, we propose Positive and Negative Prompt Supervision, which encourages negative prompts to capture inter-class features and transfers this semantic knowledge to the visual modality to enhance OOD detection performance. Our method begins with class-specific positive and negative prompts initialized by large language models (LLMs). These prompts are subsequently optimized, with positive prompts focusing on features within each class, while negative prompts highlight features around category boundaries. Additionally, a graph-based architecture is employed to aggregate semantic-aware supervision from the optimized prompt representations and propagate it to the visual branch, thereby enhancing the performance of the energy-based OOD detector. Extensive experiments on two benchmarks, CIFAR-100 and ImageNet-1K, across eight OOD datasets and five different LLMs, demonstrate that our method outperforms state-of-the-art baselines.