Rethinking Batch Sample Relationships for Data Representation: A Batch-Graph Transformer based Approach

TL;DR

This paper introduces BGFormer, a Batch-Graph Transformer that models visual and semantic relationships within mini-batches, improving robustness and effectiveness in image representation learning.

Contribution

The paper proposes a novel Batch-Graph Transformer that encodes visual and semantic sample relationships using a flexible graph model and a dual-structure self-attention mechanism.

Findings

Effective in capturing visual and semantic relationships

Robust to noisy samples due to sparse graph modeling

Improves performance on metric learning tasks

Abstract

Exploring sample relationships within each mini-batch has shown great potential for learning image representations. Existing works generally adopt the regular Transformer to model the visual content relationships, ignoring the cues of semantic/label correlations between samples. Also, they generally adopt the "full" self-attention mechanism which are obviously redundant and also sensitive to the noisy samples. To overcome these issues, in this paper, we design a simple yet flexible Batch-Graph Transformer (BGFormer) for mini-batch sample representations by deeply capturing the relationships of image samples from both visual and semantic perspectives. BGFormer has three main aspects. (1) It employs a flexible graph model, termed Batch Graph to jointly encode the visual and semantic relationships of samples within each mini-batch. (2) It explores the neighborhood relationships of samples by borrowing the idea of sparse graph representation which thus performs robustly, w.r.t., noisy samples. (3) It devises a novel Transformer architecture that mainly adopts dual structure-constrained self-attention (SSA), together with graph normalization, FFN, etc, to carefully exploit the batch graph information for sample tokens (nodes) representations. As an application, we apply BGFormer to the metric learning tasks. Extensive experiments on four popular datasets demonstrate the effectiveness of the proposed model.