DisentangleFormer: Spatial-Channel Decoupling for Multi-Channel Vision

TL;DR

DisentangleFormer introduces a spatial-channel decoupling architecture for multi-channel vision tasks, improving performance and efficiency especially in hyperspectral imaging by modeling structural and semantic cues independently.

Contribution

The paper presents a novel architecture that decouples spatial and channel processing in vision transformers, enhancing multi-channel representation learning and reducing redundancy.

Findings

Achieves state-of-the-art results on hyperspectral benchmarks.

Reduces computational cost by 17.8% in FLOPs.

Maintains competitive accuracy on ImageNet.

Abstract

Vision Transformers face a fundamental limitation: standard self-attention jointly processes spatial and channel dimensions, leading to entangled representations that prevent independent modeling of structural and semantic dependencies. This problem is especially pronounced in hyperspectral imaging, from satellite hyperspectral remote sensing to infrared pathology imaging, where channels capture distinct biophysical or biochemical cues. We propose DisentangleFormer, an architecture that achieves robust multi-channel vision representation through principled spatial-channel decoupling. Motivated by information-theoretic principles of decorrelated representation learning, our parallel design enables independent modeling of structural and semantic cues while minimizing redundancy between spatial and channel streams. Our design integrates three core components: (1) Parallel Disentanglement: Independently processes spatial-token and channel-token streams, enabling decorrelated feature learning across spatial and spectral dimensions, (2) Squeezed Token Enhancer: An adaptive calibration module that dynamically fuses spatial and channel streams, and (3) Multi-Scale FFN: complementing global attention with multi-scale local context to capture fine-grained structural and semantic dependencies. Extensive experiments on hyperspectral benchmarks demonstrate that DisentangleFormer achieves state-of-the-art performance, consistently outperforming existing models on Indian Pine, Pavia University, and Houston, the large-scale BigEarthNet remote sensing dataset, as well as an infrared pathology dataset. Moreover, it retains competitive accuracy on ImageNet while reducing computational cost by 17.8% in FLOPs. The code will be made publicly available upon acceptance.