Hybrid Routing Transformer for Zero-Shot Learning

TL;DR

This paper introduces a hybrid routing transformer (HRT) model for zero-shot learning that combines top-down and bottom-up attention with dynamic and static routing to better align visual features with semantic attributes, improving unseen class recognition.

Contribution

The paper proposes a novel transformer architecture with hybrid routing pathways that enhances semantic alignment in zero-shot learning tasks.

Findings

HRT outperforms existing methods on CUB, SUN, and AWA2 datasets.

The hybrid routing approach improves attribute-visual feature correlation.

Experimental results demonstrate the effectiveness of the proposed model.

Abstract

Zero-shot learning (ZSL) aims to learn models that can recognize unseen image semantics based on the training of data with seen semantics. Recent studies either leverage the global image features or mine discriminative local patch features to associate the extracted visual features to the semantic attributes. However, due to the lack of the necessary top-down guidance and semantic alignment for ensuring the model attending to the real attribute-correlation regions, these methods still encounter a significant semantic gap between the visual modality and the attribute modality, which makes their prediction on unseen semantics unreliable. To solve this problem, this paper establishes a novel transformer encoder-decoder model, called hybrid routing transformer (HRT). In HRT encoder, we embed an active attention, which is constructed by both the bottom-up and the top-down dynamic routing pathways to generate the attribute-aligned visual feature. While in HRT decoder, we use static routing to calculate the correlation among the attribute-aligned visual features, the corresponding attribute semantics, and the class attribute vectors to generate the final class label predictions. This design makes the presented transformer model a hybrid of 1) top-down and bottom-up attention pathways and 2) dynamic and static routing pathways. Comprehensive experiments on three widely-used benchmark datasets, namely CUB, SUN, and AWA2, are conducted. The obtained experimental results demonstrate the effectiveness of the proposed method.