Harnessing Large Language and Vision-Language Models for Robust Out-of-Distribution Detection

TL;DR

This paper introduces a novel zero-shot OOD detection method leveraging LLMs and VLMs to improve performance on both Far-OOD and Near-OOD scenarios, demonstrating significant gains and robustness across benchmarks.

Contribution

It proposes a new approach that combines LLM-generated superclasses with CLIP features and novel prompt tuning to enhance OOD detection in diverse scenarios.

Findings

Up to 2.9% improvement in AUROC

Up to 12.6% reduction in FPR95

Superior robustness against covariate shift

Abstract

Out-of-distribution (OOD) detection has seen significant advancements with zero-shot approaches by leveraging the powerful Vision-Language Models (VLMs) such as CLIP. However, prior research works have predominantly focused on enhancing Far-OOD performance, while potentially compromising Near-OOD efficacy, as observed from our pilot study. To address this issue, we propose a novel strategy to enhance zero-shot OOD detection performances for both Far-OOD and Near-OOD scenarios by innovatively harnessing Large Language Models (LLMs) and VLMs. Our approach first exploit an LLM to generate superclasses of the ID labels and their corresponding background descriptions followed by feature extraction using CLIP. We then isolate the core semantic features for ID data by subtracting background features from the superclass features. The refined representation facilitates the selection of more appropriate negative labels for OOD data from a comprehensive candidate label set of WordNet, thereby enhancing the performance of zero-shot OOD detection in both scenarios. Furthermore, we introduce novel few-shot prompt tuning and visual prompt tuning to adapt the proposed framework to better align with the target distribution. Experimental results demonstrate that the proposed approach consistently outperforms current state-of-the-art methods across multiple benchmarks, with an improvement of up to 2.9% in AUROC and a reduction of up to 12.6% in FPR95. Additionally, our method exhibits superior robustness against covariate shift across different domains, further highlighting its effectiveness in real-world scenarios.