Multi-Modal Hybrid Learning and Sequential Training for RGB-T Saliency Detection

TL;DR

This paper introduces a novel multi-modal hybrid loss and a sequential training strategy for RGB-T saliency detection, effectively fusing RGB and thermal features to improve detection accuracy in challenging scenes.

Contribution

It proposes a new hybrid loss function and a sequential training approach that enhance cross-modal feature fusion and saliency detection performance.

Findings

Outperforms existing state-of-the-art methods in RGB-T saliency detection.

Sequential training improves performance without additional computational cost.

Hybrid fusion module effectively combines spatial and channel information.

Abstract

RGB-T saliency detection has emerged as an important computer vision task, identifying conspicuous objects in challenging scenes such as dark environments. However, existing methods neglect the characteristics of cross-modal features and rely solely on network structures to fuse RGB and thermal features. To address this, we first propose a Multi-Modal Hybrid loss (MMHL) that comprises supervised and self-supervised loss functions. The supervised loss component of MMHL distinctly utilizes semantic features from different modalities, while the self-supervised loss component reduces the distance between RGB and thermal features. We further consider both spatial and channel information during feature fusion and propose the Hybrid Fusion Module to effectively fuse RGB and thermal features. Lastly, instead of jointly training the network with cross-modal features, we implement a sequential training strategy which performs training only on RGB images in the first stage and then learns cross-modal features in the second stage. This training strategy improves saliency detection performance without computational overhead. Results from performance evaluation and ablation studies demonstrate the superior performance achieved by the proposed method compared with the existing state-of-the-art methods.