Variational Probabilistic Fusion Network for RGB-T Semantic Segmentation

TL;DR

This paper introduces VPFNet, a novel probabilistic fusion network for RGB-T semantic segmentation that models fusion features as random variables, leading to more robust segmentation especially in challenging lighting conditions.

Contribution

The paper proposes a variational probabilistic fusion approach with a new VFFM module, addressing noise, class imbalance, and modality bias in RGB-T segmentation.

Findings

Achieves state-of-the-art performance on MFNet and PST900 datasets.

Effectively handles modality noise and class imbalance.

Improves robustness in poor lighting conditions.

Abstract

RGB-T semantic segmentation has been widely adopted to handle hard scenes with poor lighting conditions by fusing different modality features of RGB and thermal images. Existing methods try to find an optimal fusion feature for segmentation, resulting in sensitivity to modality noise, class-imbalance, and modality bias. To overcome the problems, this paper proposes a novel Variational Probabilistic Fusion Network (VPFNet), which regards fusion features as random variables and obtains robust segmentation by averaging segmentation results under multiple samples of fusion features. The random samples generation of fusion features in VPFNet is realized by a novel Variational Feature Fusion Module (VFFM) designed based on variation attention. To further avoid class-imbalance and modality bias, we employ the weighted cross-entropy loss and introduce prior information of illumination and category to control the proposed VFFM. Experimental results on MFNet and PST900 datasets demonstrate that the proposed VPFNet can achieve state-of-the-art segmentation performance.