Open Set Recognition using Vision Transformer with an Additional Detection Head

TL;DR

This paper introduces a novel open set recognition method using Vision Transformer with an added detection head, achieving state-of-the-art results by effectively distinguishing known from unknown classes.

Contribution

First to leverage Vision Transformer for open set recognition, employing a two-stage training process with a detection head for improved unknown class detection.

Findings

Outperforms baseline methods on multiple OSR benchmarks

Achieves new state-of-the-art performance in open set recognition

Effectively distinguishes known and unknown classes using cluster-based detection

Abstract

Deep neural networks have demonstrated prominent capacities for image classification tasks in a closed set setting, where the test data come from the same distribution as the training data. However, in a more realistic open set scenario, traditional classifiers with incomplete knowledge cannot tackle test data that are not from the training classes. Open set recognition (OSR) aims to address this problem by both identifying unknown classes and distinguishing known classes simultaneously. In this paper, we propose a novel approach to OSR that is based on the vision transformer (ViT) technique. Specifically, our approach employs two separate training stages. First, a ViT model is trained to perform closed set classification. Then, an additional detection head is attached to the embedded features extracted by the ViT, trained to force the representations of known data to class-specific clusters compactly. Test examples are identified as known or unknown based on their distance to the cluster centers. To the best of our knowledge, this is the first time to leverage ViT for the purpose of OSR, and our extensive evaluation against several OSR benchmark datasets reveals that our approach significantly outperforms other baseline methods and obtains new state-of-the-art performance.