Learning A Robust RGB-Thermal Detector for Extreme Modality Imbalance

TL;DR

This paper proposes a novel RGB-Thermal object detection framework that effectively handles extreme modality imbalance by adaptively weighing modalities, simulating real-world degradations, and enforcing consistency, leading to improved robustness and performance.

Contribution

It introduces a base-and-auxiliary detector architecture with a modality interaction module and pseudo-degradation training to address extreme modality imbalance in RGB-T detection.

Findings

Reduces Missing Rate by 55% under severe imbalance.

Improves detection robustness across various baseline models.

Effectively handles real-world modality degradation scenarios.

Abstract

RGB-Thermal (RGB-T) object detection utilizes thermal infrared (TIR) images to complement RGB data, improving robustness in challenging conditions. Traditional RGB-T detectors assume balanced training data, where both modalities contribute equally. However, in real-world scenarios, modality degradation-due to environmental factors or technical issues-can lead to extreme modality imbalance, causing out-of-distribution (OOD) issues during testing and disrupting model convergence during training. This paper addresses these challenges by proposing a novel base-and-auxiliary detector architecture. We introduce a modality interaction module to adaptively weigh modalities based on their quality and handle imbalanced samples effectively. Additionally, we leverage modality pseudo-degradation to simulate real-world imbalances in training data. The base detector, trained on high-quality pairs, provides a consistency constraint for the auxiliary detector, which receives degraded samples. This framework enhances model robustness, ensuring reliable performance even under severe modality degradation. Experimental results demonstrate the effectiveness of our method in handling extreme modality imbalances~(decreasing the Missing Rate by 55%) and improving performance across various baseline detectors.