Beyond conventional vision: RGB-event fusion for robust object detection in dynamic traffic scenarios

TL;DR

This paper introduces MCFNet, a novel RGB-event fusion network that enhances object detection in challenging traffic scenarios by integrating high dynamic range event data with RGB images through advanced alignment and fusion modules.

Contribution

The paper proposes a new motion cue fusion network (MCFNet) with modules for event correction, dynamic upsampling, and adaptive cross-modal fusion, improving detection robustness in poor lighting and fast motion conditions.

Findings

MCFNet outperforms existing methods on DSEC-Det and PKU-DAVIS-SOD datasets.

Achieves 7.4% higher mAP50 on DSEC-Det.

Demonstrates robustness in nighttime and tunnel scenarios.

Abstract

The dynamic range limitation of conventional RGB cameras reduces global contrast and causes loss of high-frequency details such as textures and edges in complex traffic environments (e.g., nighttime driving, tunnels), hindering discriminative feature extraction and degrading frame-based object detection. To address this, we integrate a bio-inspired event camera with an RGB camera to provide high dynamic range information and propose a motion cue fusion network (MCFNet), which achieves optimal spatiotemporal alignment and adaptive cross-modal feature fusion under challenging lighting. Specifically, an event correction module (ECM) temporally aligns asynchronous event streams with image frames via optical-flow-based warping, jointly optimized with the detection network to learn task-aware event representations. The event dynamic upsampling module (EDUM) enhances spatial resolution of event frames to match image structures, ensuring precise spatiotemporal alignment. The cross-modal mamba fusion module (CMM) uses adaptive feature fusion with a novel interlaced scanning mechanism, effectively integrating complementary information for robust detection. Experiments conducted on the DSEC-Det and PKU-DAVIS-SOD datasets demonstrate that MCFNet significantly outperforms existing methods in various poor lighting and fast moving traffic scenarios. Notably, on the DSEC-Det dataset, MCFNet achieves a remarkable improvement, surpassing the best existing methods by 7.4% in mAP50 and 1.7% in mAP metrics, respectively. The code is available at https://github.com/Charm11492/MCFNet.