RTFDNet: Fusion-Decoupling for Robust RGB-T Segmentation

TL;DR

RTFDNet introduces a unified fusion-decoupling framework for RGB-T segmentation, improving robustness and performance in low-light conditions by effectively isolating modality-specific features and enhancing unimodal and fused representations.

Contribution

The paper proposes RTFDNet, a novel three-branch encoder-decoder that unifies fusion and decoupling for robust RGB-T segmentation, enabling efficient inference and improved modality handling.

Findings

Consistent performance across varying modality conditions.

Effective isolation of modality-specific features improves robustness.

Enables standalone inference without performance loss.

Abstract

RGB-Thermal (RGB-T) semantic segmentation is essential for robotic systems operating in low-light or dark environments. However, traditional approaches often overemphasize modality balance, resulting in limited robustness and severe performance degradation when sensor signals are partially missing. Recent advances such as cross-modal knowledge distillation and modality-adaptive fine-tuning attempt to enhance cross-modal interaction, but they typically decouple modality fusion and modality adaptation, requiring multi-stage training with frozen models or teacher-student frameworks. We present RTFDNet, a three-branch encoder-decoder that unifies fusion and decoupling for robust RGB-T segmentation. Synergistic Feature Fusion (SFF) performs channel-wise gated exchange and lightweight spatial attention to inject complementary cues. Cross-Modal Decouple Regularization (CMDR) isolates modality-specific components from the fused representation and supervises unimodal decoders via stop-gradient targets. Region Decouple Regularization (RDR) enforces class-selective prediction consistency in confident regions while blocking gradients to the fusion branch. This feedback loop strengthens unimodal paths without degrading the fused stream, enabling efficient standalone inference at test time. Extensive experiments demonstrate the effectiveness of RTFDNet, showing consistent performance across varying modality conditions. Our implementation will be released to facilitate further research. Our source code are publicly available at https://github.com/curapima/RTFDNet.