Dual Relation Mining Network for Zero-Shot Learning

TL;DR

This paper introduces a Dual Relation Mining Network (DRMN) that improves zero-shot learning by enhancing visual-semantic interactions and modeling attribute relationships, achieving state-of-the-art results on standard benchmarks.

Contribution

The paper proposes a novel DRMN framework with dual attention and semantic interaction transformer to better capture visual-semantic and attribute relationships in ZSL.

Findings

Achieves new state-of-the-art on CUB, SUN, and AwA2 benchmarks.

Effectively models semantic attribute relationships with a transformer.

Enhances visual-semantic alignment through dual attention mechanisms.

Abstract

Zero-shot learning (ZSL) aims to recognize novel classes through transferring shared semantic knowledge (e.g., attributes) from seen classes to unseen classes. Recently, attention-based methods have exhibited significant progress which align visual features and attributes via a spatial attention mechanism. However, these methods only explore visual-semantic relationship in the spatial dimension, which can lead to classification ambiguity when different attributes share similar attention regions, and semantic relationship between attributes is rarely discussed. To alleviate the above problems, we propose a Dual Relation Mining Network (DRMN) to enable more effective visual-semantic interactions and learn semantic relationship among attributes for knowledge transfer. Specifically, we introduce a Dual Attention Block (DAB) for visual-semantic relationship mining, which enriches visual information by multi-level feature fusion and conducts spatial attention for visual to semantic embedding. Moreover, an attribute-guided channel attention is utilized to decouple entangled semantic features. For semantic relationship modeling, we utilize a Semantic Interaction Transformer (SIT) to enhance the generalization of attribute representations among images. Additionally, a global classification branch is introduced as a complement to human-defined semantic attributes, and we then combine the results with attribute-based classification. Extensive experiments demonstrate that the proposed DRMN leads to new state-of-the-art performances on three standard ZSL benchmarks, i.e., CUB, SUN, and AwA2.