BOAT: Bilateral Local Attention Vision Transformer

TL;DR

BOAT introduces a novel vision transformer that combines local attention in both image space and feature space, enhancing the ability to capture distant relationships while maintaining efficiency.

Contribution

The paper proposes a bilateral local attention mechanism that integrates feature-space and image-space attention, improving the modeling of long-range dependencies in vision transformers.

Findings

BOAT-CSWin outperforms existing state-of-the-art models on benchmark datasets.

The bilateral attention mechanism improves capturing distant patch relationships.

Extensive experiments validate the effectiveness of the proposed method.

Abstract

Vision Transformers achieved outstanding performance in many computer vision tasks. Early Vision Transformers such as ViT and DeiT adopt global self-attention, which is computationally expensive when the number of patches is large. To improve efficiency, recent Vision Transformers adopt local self-attention mechanisms, where self-attention is computed within local windows. Despite the fact that window-based local self-attention significantly boosts efficiency, it fails to capture the relationships between distant but similar patches in the image plane. To overcome this limitation of image-space local attention, in this paper, we further exploit the locality of patches in the feature space. We group the patches into multiple clusters using their features, and self-attention is computed within every cluster. Such feature-space local attention effectively captures the connections between patches across different local windows but still relevant. We propose a Bilateral lOcal Attention vision Transformer (BOAT), which integrates feature-space local attention with image-space local attention. We further integrate BOAT with both Swin and CSWin models, and extensive experiments on several benchmark datasets demonstrate that our BOAT-CSWin model clearly and consistently outperforms existing state-of-the-art CNN models and vision Transformers.