Spectral-Aware Global Fusion for RGB-Thermal Semantic Segmentation

TL;DR

This paper introduces SGFNet, a spectral-aware fusion network that improves RGB-thermal semantic segmentation by explicitly modeling high- and low-frequency feature interactions, leading to better performance in challenging conditions.

Contribution

The paper proposes a novel spectral perspective and a fusion network that explicitly models high- and low-frequency feature interactions for RGB-thermal segmentation.

Findings

SGFNet outperforms state-of-the-art methods on MFNet and PST900 datasets.

Explicit spectral modeling enhances multi-modal feature fusion.

The approach improves robustness in low-light and obscured scenarios.

Abstract

Semantic segmentation relying solely on RGB data often struggles in challenging conditions such as low illumination and obscured views, limiting its reliability in critical applications like autonomous driving. To address this, integrating additional thermal radiation data with RGB images demonstrates enhanced performance and robustness. However, how to effectively reconcile the modality discrepancies and fuse the RGB and thermal features remains a well-known challenge. In this work, we address this challenge from a novel spectral perspective. We observe that the multi-modal features can be categorized into two spectral components: low-frequency features that provide broad scene context, including color variations and smooth areas, and high-frequency features that capture modality-specific details such as edges and textures. Inspired by this, we propose the Spectral-aware Global Fusion Network (SGFNet) to effectively enhance and fuse the multi-modal features by explicitly modeling the interactions between the high-frequency, modality-specific features. Our experimental results demonstrate that SGFNet outperforms the state-of-the-art methods on the MFNet and PST900 datasets.