Modality-Decoupled RGB-Thermal Object Detector via Query Fusion

TL;DR

This paper introduces a novel RGB-T object detection framework that effectively balances modality fusion and separation, improving robustness under poor-quality conditions and enabling training with unpaired data.

Contribution

The proposed MDQF framework employs separate branches with query fusion for modality decoupling, allowing improved detection robustness and unpaired data training, surpassing existing methods.

Findings

Outperforms existing RGB-T detectors in accuracy.

Enhances modality independence and robustness.

Allows training with unpaired RGB or TIR images.

Abstract

The advantage of RGB-Thermal (RGB-T) detection lies in its ability to perform modality fusion and integrate cross-modality complementary information, enabling robust detection under diverse illumination and weather conditions. However, under extreme conditions where one modality exhibits poor quality and disturbs detection, modality separation is necessary to mitigate the impact of noise. To address this problem, we propose a Modality-Decoupled RGB-T detection framework with Query Fusion (MDQF) to balance modality complementation and separation. In this framework, DETR-like detectors are employed as separate branches for the RGB and TIR images, with query fusion interspersed between the two branches in each refinement stage. Herein, query fusion is performed by feeding the high-quality queries from one branch to the other one after query selection and adaptation. This design effectively excludes the degraded modality and corrects the predictions using high-quality queries. Moreover, the decoupled framework allows us to optimize each individual branch with unpaired RGB or TIR images, eliminating the need for paired RGB-T data. Extensive experiments demonstrate that our approach delivers superior performance to existing RGB-T detectors and achieves better modality independence.