Multi-Head Self-Attention via Vision Transformer for Zero-Shot Learning

TL;DR

This paper introduces an attention-based Vision Transformer model for Zero-Shot Learning that effectively captures discriminative attributes from images, achieving state-of-the-art results on multiple benchmarks.

Contribution

It proposes a novel attention mechanism adapted from Vision Transformer to explicitly localize and learn attributes for unseen class recognition in ZSL.

Findings

Achieved new state-of-the-art harmonic mean results on AWA2, CUB, and SUN datasets.

Demonstrated the effectiveness of attribute localization via attention in ZSL.

Validated the approach's superiority over existing methods.

Abstract

Zero-Shot Learning (ZSL) aims to recognise unseen object classes, which are not observed during the training phase. The existing body of works on ZSL mostly relies on pretrained visual features and lacks the explicit attribute localisation mechanism on images. In this work, we propose an attention-based model in the problem settings of ZSL to learn attributes useful for unseen class recognition. Our method uses an attention mechanism adapted from Vision Transformer to capture and learn discriminative attributes by splitting images into small patches. We conduct experiments on three popular ZSL benchmarks (i.e., AWA2, CUB and SUN) and set new state-of-the-art harmonic mean results {on all the three datasets}, which illustrate the effectiveness of our proposed method.