LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models

TL;DR

LAPT introduces an automated prompt tuning method for vision-language models that enhances out-of-distribution detection, reduces manual effort, and improves robustness and accuracy across diverse OOD scenarios.

Contribution

The paper presents a novel automated prompt tuning framework that eliminates manual prompt engineering for OOD detection with vision-language models, using distribution-aware prompts and autonomous data collection.

Findings

LAPT outperforms manual prompts in OOD detection accuracy.

It improves ID classification and robustness to covariate shifts.

The method operates autonomously with only class names as input.

Abstract

Out-of-distribution (OOD) detection is crucial for model reliability, as it identifies samples from unknown classes and reduces errors due to unexpected inputs. Vision-Language Models (VLMs) such as CLIP are emerging as powerful tools for OOD detection by integrating multi-modal information. However, the practical application of such systems is challenged by manual prompt engineering, which demands domain expertise and is sensitive to linguistic nuances. In this paper, we introduce Label-driven Automated Prompt Tuning (LAPT), a novel approach to OOD detection that reduces the need for manual prompt engineering. We develop distribution-aware prompts with in-distribution (ID) class names and negative labels mined automatically. Training samples linked to these class labels are collected autonomously via image synthesis and retrieval methods, allowing for prompt learning without manual effort. We utilize a simple cross-entropy loss for prompt optimization, with cross-modal and cross-distribution mixing strategies to reduce image noise and explore the intermediate space between distributions, respectively. The LAPT framework operates autonomously, requiring only ID class names as input and eliminating the need for manual intervention. With extensive experiments, LAPT consistently outperforms manually crafted prompts, setting a new standard for OOD detection. Moreover, LAPT not only enhances the distinction between ID and OOD samples, but also improves the ID classification accuracy and strengthens the generalization robustness to covariate shifts, resulting in outstanding performance in challenging full-spectrum OOD detection tasks. Codes are available at \url{https://github.com/YBZh/LAPT}.