Modality Dominance-Aware Optimization for Embodied RGB-Infrared Perception

TL;DR

This paper introduces a novel framework, MDACL, that addresses the imbalance in optimization between RGB and infrared modalities in perception systems, leading to improved fusion and detection performance.

Contribution

The paper proposes the Modality Dominance Index (MDI) and a new MDACL framework with HCG and AER to balance cross-modal optimization in RGB-IR perception.

Findings

MDACL achieves state-of-the-art results on three RGB-IR benchmarks.

MDACL effectively reduces modality dominance bias during training.

The approach improves feature alignment and detection accuracy.

Abstract

RGB-Infrared (RGB-IR) multimodal perception is fundamental to embodied multimedia systems operating in complex physical environments. Although recent cross-modal fusion methods have advanced RGB-IR detection, the optimization dynamics caused by asymmetric modality characteristics remain underexplored. In practice, disparities in information density and feature quality introduce persistent optimization bias, leading training to overemphasize a dominant modality and hindering effective fusion. To quantify this phenomenon, we propose the Modality Dominance Index (MDI), which measures modality dominance by jointly modeling feature entropy and gradient contribution. Based on MDI, we develop a Modality Dominance-Aware Cross-modal Learning (MDACL) framework that regulates cross-modal optimization. MDACL incorporates Hierarchical Cross-modal Guidance (HCG) to enhance feature alignment and Adversarial Equilibrium Regularization (AER) to balance optimization dynamics during fusion. Extensive experiments on three RGB-IR benchmarks demonstrate that MDACL effectively mitigates optimization bias and achieves SOTA performance.