Frequency-Dynamic Attention Modulation for Dense Prediction

TL;DR

This paper introduces Frequency-Dynamic Attention Modulation (FDAM), a novel method inspired by circuit theory, to enhance Vision Transformers by dynamically adjusting their frequency response, thereby preserving details and textures across various vision tasks.

Contribution

FDAM is a new, plug-in strategy for ViTs that modulates frequency response using attention inversion and dynamic scaling, improving performance without representation collapse.

Findings

Improves semantic segmentation, object detection, and instance segmentation results.

Achieves state-of-the-art remote sensing detection performance.

Avoids representation collapse through feature similarity and rank analysis.

Abstract

Vision Transformers (ViTs) have significantly advanced computer vision, demonstrating strong performance across various tasks. However, the attention mechanism in ViTs makes each layer function as a low-pass filter, and the stacked-layer architecture in existing transformers suffers from frequency vanishing. This leads to the loss of critical details and textures. We propose a novel, circuit-theory-inspired strategy called Frequency-Dynamic Attention Modulation (FDAM), which can be easily plugged into ViTs. FDAM directly modulates the overall frequency response of ViTs and consists of two techniques: Attention Inversion (AttInv) and Frequency Dynamic Scaling (FreqScale). Since circuit theory uses low-pass filters as fundamental elements, we introduce AttInv, a method that generates complementary high-pass filtering by inverting the low-pass filter in the attention matrix, and dynamically combining the two. We further design FreqScale to weight different frequency components for fine-grained adjustments to the target response function. Through feature similarity analysis and effective rank evaluation, we demonstrate that our approach avoids representation collapse, leading to consistent performance improvements across various models, including SegFormer, DeiT, and MaskDINO. These improvements are evident in tasks such as semantic segmentation, object detection, and instance segmentation. Additionally, we apply our method to remote sensing detection, achieving state-of-the-art results in single-scale settings. The code is available at https://github.com/Linwei-Chen/FDAM.