EvRT-DETR: Latent Space Adaptation of Image Detectors for Event-based Vision

TL;DR

This paper introduces EvRT-DETR, a novel framework that adapts mainstream image detectors for event-based cameras, achieving state-of-the-art object detection performance with minimal architectural modifications.

Contribution

It presents a new adaptation technique that transforms image-based detectors into event-based models by modifying their latent space, enabling efficient and effective EBC object detection.

Findings

Achieves state-of-the-art performance on Gen1 and 1Mpx/Gen4 datasets.

Demonstrates that a simple image-like representation with RT-DETR is competitive.

Provides an efficient adaptation method with minimal architectural changes.

Abstract

Event-based cameras (EBCs) have emerged as a bio-inspired alternative to traditional cameras, offering advantages in power efficiency, temporal resolution, and high dynamic range. However, the development of image analysis methods for EBCs is challenging due to the sparse and asynchronous nature of the data. This work addresses the problem of object detection for EBC cameras. The current approaches to EBC object detection focus on constructing complex data representations and rely on specialized architectures. We introduce I2EvDet (Image-to-Event Detection), a novel adaptation framework that bridges mainstream object detection with temporal event data processing. First, we demonstrate that a Real-Time DEtection TRansformer, or RT-DETR, a state-of-the-art natural image detector, trained on a simple image-like representation of the EBC data achieves performance comparable to specialized EBC methods. Next, as part of our framework, we develop an efficient adaptation technique that transforms image-based detectors into event-based detection models by modifying their frozen latent representation space through minimal architectural additions. The resulting EvRT-DETR model reaches state-of-the-art performance on the standard benchmark datasets Gen1 (mAP $+2.3$) and 1Mpx/Gen4 (mAP $+1.4$). These results demonstrate a fundamentally new approach to EBC object detection through principled adaptation of mainstream architectures, offering an efficient alternative with potential applications to other temporal visual domains. The code is available at: https://github.com/realtime-intelligence/evrt-detr