DiffPixelFormer: Differential Pixel-Aware Transformer for RGB-D Indoor Scene Segmentation

TL;DR

DiffPixelFormer introduces a novel differential pixel-aware Transformer that improves RGB-D indoor scene segmentation by enhancing intra-modal features and modeling inter-modal interactions for more accurate pixel-level alignment.

Contribution

It proposes the Intra-Inter Modal Interaction Block and a dynamic fusion strategy to better model intra- and inter-modal relationships in RGB-D segmentation.

Findings

Achieves state-of-the-art mIoU scores on SUN RGB-D and NYUDv2 datasets.

Outperforms previous methods like DFormer-L by significant margins.

Demonstrates effective pixel-level cross-modal alignment and scene understanding.

Abstract

Indoor semantic segmentation is fundamental to computer vision and robotics, supporting applications such as autonomous navigation, augmented reality, and smart environments. Although RGB-D fusion leverages complementary appearance and geometric cues, existing methods often depend on computationally intensive cross-attention mechanisms and insufficiently model intra- and inter-modal feature relationships, resulting in imprecise feature alignment and limited discriminative representation. To address these challenges, we propose DiffPixelFormer, a differential pixel-aware Transformer for RGB-D indoor scene segmentation that simultaneously enhances intra-modal representations and models inter-modal interactions. At its core, the Intra-Inter Modal Interaction Block (IIMIB) captures intra-modal long-range dependencies via self-attention and models inter-modal interactions with the Differential-Shared Inter-Modal (DSIM) module to disentangle modality-specific and shared cues, enabling fine-grained, pixel-level cross-modal alignment. Furthermore, a dynamic fusion strategy balances modality contributions and fully exploits RGB-D information according to scene characteristics. Extensive experiments on the SUN RGB-D and NYUDv2 benchmarks demonstrate that DiffPixelFormer-L achieves mIoU scores of 54.28% and 59.95%, outperforming DFormer-L by 1.78% and 2.75%, respectively. Code is available at https://github.com/gongyan1/DiffPixelFormer.