Toward Deep Representation Learning for Event-Enhanced Visual Autonomous Perception: the eAP Dataset

TL;DR

This paper introduces the eAP dataset, a large-scale event camera dataset for autonomous perception, enabling improved deep learning models for vehicle detection and TTC estimation in challenging lighting conditions.

Contribution

The paper presents the first large-scale event camera dataset for autonomous driving, facilitating research in event-enhanced perception tasks.

Findings

Events improve vehicle detection in low-light scenarios

Geometry-aware learning enhances TTC estimation accuracy

Event-based models operate at 200 FPS

Abstract

Recent visual autonomous perception systems achieve remarkable performances with deep representation learning. However, they fail in scenarios with challenging illumination.While event cameras can mitigate this problem, there is a lack of a large-scale dataset to develop event-enhanced deep visual perception models in autonomous driving scenes. To address the gap, we present the eAP (event-enhanced Autonomous Perception) dataset, the largest dataset with event cameras for autonomous perception. We demonstrate how eAP can facilitate the study of different autonomous perception tasks, including 3D vehicle detection and object time-to-contact (TTC) estimation, through deep representation learning. Based on eAP, we demonstrate the ffrst successful use of events to improve a popular 3D vehicle detection network in challenging illumination scenarios. eAP also enables a devoted study of the representation learning problem of object TTC estimation. We show how a geometryaware representation learning framework leads to the best eventbased object TTC estimation network that operates at 200 FPS. The dataset, code, and pre-trained models will be made publicly available for future research.