Contrastive Out-of-Distribution Detection for Pretrained Transformers

TL;DR

This paper introduces an unsupervised contrastive fine-tuning method for pretrained Transformers that significantly enhances out-of-distribution detection by producing more compact representations, enabling near-perfect OOD identification.

Contribution

It proposes a novel contrastive learning approach for Transformers that improves OOD detection without requiring labeled OOD data, outperforming existing methods.

Findings

Achieves near-perfect OOD detection performance.

Contrastive fine-tuning produces more compact representations.

Outperforms baseline OOD detection methods.

Abstract

Pretrained Transformers achieve remarkable performance when training and test data are from the same distribution. However, in real-world scenarios, the model often faces out-of-distribution (OOD) instances that can cause severe semantic shift problems at inference time. Therefore, in practice, a reliable model should identify such instances, and then either reject them during inference or pass them over to models that handle another distribution. In this paper, we develop an unsupervised OOD detection method, in which only the in-distribution (ID) data are used in training. We propose to fine-tune the Transformers with a contrastive loss, which improves the compactness of representations, such that OOD instances can be better differentiated from ID ones. These OOD instances can then be accurately detected using the Mahalanobis distance in the model's penultimate layer. We experiment with comprehensive settings and achieve near-perfect OOD detection performance, outperforming baselines drastically. We further investigate the rationales behind the improvement, finding that more compact representations through margin-based contrastive learning bring the improvement. We release our code to the community for future research.