Dynamic Granularity Matters: Rethinking Vision Transformers Beyond Fixed Patch Splitting

TL;DR

Grc-ViT introduces a dynamic, adaptive approach to vision transformers that adjusts visual granularity based on image complexity, improving fine-grained detail capture and efficiency.

Contribution

It proposes a novel coarse-to-fine framework with learnable parameters for adaptive granularity, surpassing fixed patch methods in accuracy and efficiency.

Findings

Improves fine-grained discrimination in vision transformers.

Achieves better accuracy-efficiency trade-off than fixed patch methods.

Demonstrates effectiveness across multiple image complexity scenarios.

Abstract

Vision Transformers (ViTs) have demonstrated strong capabilities in capturing global dependencies but often struggle to efficiently represent fine-grained local details. Existing multi-scale approaches alleviate this issue by integrating hierarchical or hybrid features; however, they rely on fixed patch sizes and introduce redundant computation. To address these limitations, we propose Granularity-driven Vision Transformer (Grc-ViT), a dynamic coarse-to-fine framework that adaptively adjusts visual granularity based on image complexity. It comprises two key stages: (1) Coarse Granularity Evaluation module, which assesses visual complexity using edge density, entropy, and frequency-domain cues to estimate suitable patch and window sizes; (2) Fine-grained Refinement module, which refines attention computation according to the selected granularity, enabling efficient and precise feature learning. Two learnable parameters, {\alpha} and \b{eta}, are optimized end-to-end to balance global reasoning and local perception. Comprehensive evaluations demonstrate that Grc-ViT enhances fine-grained discrimination while achieving a superior trade-off between accuracy and computational efficiency.