Learning Cross-Image Object Semantic Relation in Transformer for Few-Shot Fine-Grained Image Classification

TL;DR

This paper introduces HelixFormer, a Transformer-based model that effectively mines cross-image semantic relations to improve few-shot fine-grained image classification, outperforming existing methods on multiple benchmarks.

Contribution

The paper proposes a novel double-helix Transformer architecture with relation mining and representation enhancement processes for better cross-image semantic relation modeling in few-shot learning.

Findings

HelixFormer achieves superior accuracy on five fine-grained benchmarks.

The model outperforms state-of-the-art methods in 1-shot and 5-shot scenarios.

Extensive experiments validate the effectiveness of cross-image relation mining.

Abstract

Few-shot fine-grained learning aims to classify a query image into one of a set of support categories with fine-grained differences. Although learning different objects' local differences via Deep Neural Networks has achieved success, how to exploit the query-support cross-image object semantic relations in Transformer-based architecture remains under-explored in the few-shot fine-grained scenario. In this work, we propose a Transformer-based double-helix model, namely HelixFormer, to achieve the cross-image object semantic relation mining in a bidirectional and symmetrical manner. The HelixFormer consists of two steps: 1) Relation Mining Process (RMP) across different branches, and 2) Representation Enhancement Process (REP) within each individual branch. By the designed RMP, each branch can extract fine-grained object-level Cross-image Semantic Relation Maps (CSRMs) using information from the other branch, ensuring better cross-image interaction in semantically related local object regions. Further, with the aid of CSRMs, the developed REP can strengthen the extracted features for those discovered semantically-related local regions in each branch, boosting the model's ability to distinguish subtle feature differences of fine-grained objects. Extensive experiments conducted on five public fine-grained benchmarks demonstrate that HelixFormer can effectively enhance the cross-image object semantic relation matching for recognizing fine-grained objects, achieving much better performance over most state-of-the-art methods under 1-shot and 5-shot scenarios. Our code is available at: https://github.com/JiakangYuan/HelixFormer