EvPlug: Learn a Plug-and-Play Module for Event and Image Fusion

TL;DR

EvPlug introduces a plug-and-play fusion module that integrates event camera data with RGB images, enhancing robustness and temporal resolution in vision tasks without modifying existing models or requiring labeled event data.

Contribution

It presents a novel, training-free fusion module that can be added to RGB models to incorporate event data, improving performance in HDR and fast motion scenarios.

Findings

Outperforms baseline models in object detection, segmentation, and 3D hand pose estimation.

Requires only unlabeled event-image pairs for training.

Does not alter the original RGB model structure or weights.

Abstract

Event cameras and RGB cameras exhibit complementary characteristics in imaging: the former possesses high dynamic range (HDR) and high temporal resolution, while the latter provides rich texture and color information. This makes the integration of event cameras into middle- and high-level RGB-based vision tasks highly promising. However, challenges arise in multi-modal fusion, data annotation, and model architecture design. In this paper, we propose EvPlug, which learns a plug-and-play event and image fusion module from the supervision of the existing RGB-based model. The learned fusion module integrates event streams with image features in the form of a plug-in, endowing the RGB-based model to be robust to HDR and fast motion scenes while enabling high temporal resolution inference. Our method only requires unlabeled event-image pairs (no pixel-wise alignment required) and does not alter the structure or weights of the RGB-based model. We demonstrate the superiority of EvPlug in several vision tasks such as object detection, semantic segmentation, and 3D hand pose estimation