Knowledge Regularized Negative Feature Tuning of Vision-Language Models for Out-of-Distribution Detection

TL;DR

KR-NFT introduces a knowledge-regularized feature tuning method that enhances out-of-distribution detection in vision-language models by separating features and dynamically adapting to images, outperforming traditional methods especially with limited data.

Contribution

The paper proposes a novel Knowledge Regularized Negative Feature Tuning (KR-NFT) method that improves OOD detection and ID classification by separating features and using adaptive, knowledge-regularized optimization.

Findings

KR-NFT outperforms traditional negative prompt tuning in efficiency and scalability.

It significantly reduces false positive rate (FPR95) by 5.44% with few-shot ImageNet training.

The method enhances OOD detection on unseen ID datasets while maintaining ID classification accuracy.

Abstract

Out-of-distribution (OOD) detection is crucial for building reliable machine learning models. Although negative prompt tuning has enhanced the OOD detection capabilities of vision-language models, these tuned models often suffer from reduced generalization performance on unseen classes and styles. To address this challenge, we propose a novel method called Knowledge Regularized Negative Feature Tuning (KR-NFT), which integrates an innovative adaptation architecture termed Negative Feature Tuning (NFT) and a corresponding knowledge-regularization (KR) optimization strategy. Specifically, NFT applies distribution-aware transformations to pre-trained text features, effectively separating positive and negative features into distinct spaces. This separation maximizes the distinction between in-distribution (ID) and OOD images. Additionally, we introduce image-conditional learnable factors through a lightweight meta-network, enabling dynamic adaptation to individual images and mitigating sensitivity to class and style shifts. Compared to traditional negative prompt tuning, NFT demonstrates superior efficiency and scalability. To optimize this adaptation architecture, the KR optimization strategy is designed to enhance the discrimination between ID and OOD sets while mitigating pre-trained knowledge forgetting. This enhances OOD detection performance on trained ID classes while simultaneously improving OOD detection on unseen ID datasets. Notably, when trained with few-shot samples from ImageNet dataset, KR-NFT not only improves ID classification accuracy and OOD detection but also significantly reduces the FPR95 by 5.44\% under an unexplored generalization setting with unseen ID categories. Codes can be found at \href{https://github.com/ZhuWenjie98/KRNFT}.