PViT: Prior-augmented Vision Transformer for Out-of-distribution Detection

TL;DR

PViT is a novel framework that enhances Vision Transformer's robustness for out-of-distribution detection by leveraging prior class logits, significantly outperforming existing methods on large-scale benchmarks without extra data modeling.

Contribution

Introduces PViT, a generic prior-augmented Vision Transformer that improves OOD detection by using prior logits, without additional data modeling or structural changes.

Findings

PViT outperforms SOTA methods on ImageNet benchmark.

PViT achieves lower FPR95 and higher AUROC scores.

The approach does not require extra data generation or structural modifications.

Abstract

Vision Transformers (ViTs) have achieved remarkable success over various vision tasks, yet their robustness against data distribution shifts and inherent inductive biases remain underexplored. To enhance the robustness of ViT models for image Out-of-Distribution (OOD) detection, we introduce a novel and generic framework named Prior-augmented Vision Transformer (PViT). Taking as input the prior class logits from a pretrained model, we train PViT to predict the class logits. During inference, PViT identifies OOD samples by quantifying the divergence between the predicted class logits and the prior logits obtained from pre-trained models. Unlike existing state-of-the-art(SOTA) OOD detection methods, PViT shapes the decision boundary between ID and OOD by utilizing the proposed prior guided confidence, without requiring additional data modeling, generation methods, or structural modifications. Extensive experiments on the large-scale ImageNet benchmark, evaluated against over seven OOD datasets, demonstrate that PViT significantly outperforms existing SOTA OOD detection methods in terms of FPR95 and AUROC. The codebase is publicly available at https://github.com/RanchoGoose/PViT.