M2Former: Multi-Scale Patch Selection for Fine-Grained Visual Recognition

TL;DR

M2Former introduces multi-scale patch selection and cross-scale interaction mechanisms to enhance fine-grained visual recognition, outperforming existing CNN and ViT models on standard benchmarks.

Contribution

The paper proposes a novel multi-scale patch selection method with cross-scale interactions for ViT-based FGVR, improving representational richness and scale robustness.

Findings

Outperforms CNN and ViT models on FGVR benchmarks.

Enhances recognition of objects across various sizes.

Improves feature hierarchy and model performance.

Abstract

Recently, vision Transformers (ViTs) have been actively applied to fine-grained visual recognition (FGVR). ViT can effectively model the interdependencies between patch-divided object regions through an inherent self-attention mechanism. In addition, patch selection is used with ViT to remove redundant patch information and highlight the most discriminative object patches. However, existing ViT-based FGVR models are limited to single-scale processing, and their fixed receptive fields hinder representational richness and exacerbate vulnerability to scale variability. Therefore, we propose multi-scale patch selection (MSPS) to improve the multi-scale capabilities of existing ViT-based models. Specifically, MSPS selects salient patches of different scales at different stages of a multi-scale vision Transformer (MS-ViT). In addition, we introduce class token transfer (CTT) and multi-scale cross-attention (MSCA) to model cross-scale interactions between selected multi-scale patches and fully reflect them in model decisions. Compared to previous single-scale patch selection (SSPS), our proposed MSPS encourages richer object representations based on feature hierarchy and consistently improves performance from small-sized to large-sized objects. As a result, we propose M2Former, which outperforms CNN-/ViT-based models on several widely used FGVR benchmarks.