Retrieval-Augmented Prompt for OOD Detection

TL;DR

This paper introduces Retrieval-Augmented Prompt (RAP), a novel method that enhances out-of-distribution detection by retrieving external knowledge to improve semantic supervision and adapt prompts in real-time, achieving state-of-the-art results.

Contribution

RAP leverages external knowledge retrieval to dynamically augment prompts for improved OOD detection, addressing limitations of existing methods that rely on limited or mismatched outlier data.

Findings

RAP achieves state-of-the-art performance on large-scale OOD benchmarks.

In 1-shot OOD detection on ImageNet-1k, RAP reduces FPR95 by 7.05%.

RAP improves AUROC by 1.71% over previous methods.

Abstract

Out-of-Distribution (OOD) detection is crucial for the reliable deployment of machine learning models in-the-wild, enabling accurate identification of test samples that differ from the training data distribution. Existing methods rely on auxiliary outlier samples or in-distribution (ID) data to generate outlier information for training, but due to limited outliers and their mismatch with real test OOD samples, they often fail to provide sufficient semantic supervision, leading to suboptimal performance. To address this, we propose a novel OOD detection method called Retrieval-Augmented Prompt (RAP). RAP augments a pre-trained vision-language model's prompts by retrieving external knowledge, offering enhanced semantic supervision for OOD detection. During training, RAP retrieves descriptive words for outliers based on joint similarity with external textual knowledge and uses them to augment the model's OOD prompts. During testing, RAP dynamically updates OOD prompts in real-time based on the encountered OOD samples, enabling the model to rapidly adapt to the test environment. Our extensive experiments demonstrate that RAP achieves state-of-the-art performance on large-scale OOD detection benchmarks. For example, in 1-shot OOD detection on the ImageNet-1k dataset, RAP reduces the average FPR95 by 7.05% and improves the AUROC by 1.71% compared to previous methods. Additionally, comprehensive ablation studies validate the effectiveness of each module and the underlying motivations of our approach.