I2DFormer: Learning Image to Document Attention for Zero-Shot Image Classification

TL;DR

I2DFormer introduces a transformer-based framework that leverages online textual documents to improve zero-shot image classification by aligning images and documents in a shared space, enhancing interpretability and performance.

Contribution

The paper proposes a novel cross-modal attention module and a transformer-based approach that jointly encodes images and documents for improved zero-shot learning without human-annotated attributes.

Findings

Outperforms previous unsupervised semantic embeddings in ZSL tasks

Learns discriminative document embeddings capturing visual similarities

Enables grounding of document words in image regions

Abstract

Despite the tremendous progress in zero-shot learning(ZSL), the majority of existing methods still rely on human-annotated attributes, which are difficult to annotate and scale. An unsupervised alternative is to represent each class using the word embedding associated with its semantic class name. However, word embeddings extracted from pre-trained language models do not necessarily capture visual similarities, resulting in poor zero-shot performance. In this work, we argue that online textual documents, e.g., Wikipedia, contain rich visual descriptions about object classes, therefore can be used as powerful unsupervised side information for ZSL. To this end, we propose I2DFormer, a novel transformer-based ZSL framework that jointly learns to encode images and documents by aligning both modalities in a shared embedding space. In order to distill discriminative visual words from noisy documents, we introduce a new cross-modal attention module that learns fine-grained interactions between image patches and document words. Consequently, our I2DFormer not only learns highly discriminative document embeddings that capture visual similarities but also gains the ability to localize visually relevant words in image regions. Quantitatively, we demonstrate that our I2DFormer significantly outperforms previous unsupervised semantic embeddings under both zero-shot and generalized zero-shot learning settings on three public datasets. Qualitatively, we show that our method leads to highly interpretable results where document words can be grounded in the image regions.