Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks

TL;DR

This paper provides a comprehensive survey of deep learning techniques applied to event-based vision, including representations, methods, benchmarks, and future challenges, highlighting the field's recent advancements and open problems.

Contribution

It offers the first structured taxonomy of deep learning methods for event-based vision and benchmarks key tasks like reconstruction and recognition.

Findings

Deep learning significantly improves event-based image reconstruction.

Benchmark results reveal current methods' strengths and limitations.

Identifies challenges and future directions in event-based deep learning.

Abstract

Event cameras are bio-inspired sensors that capture the per-pixel intensity changes asynchronously and produce event streams encoding the time, pixel position, and polarity (sign) of the intensity changes. Event cameras possess a myriad of advantages over canonical frame-based cameras, such as high temporal resolution, high dynamic range, low latency, etc. Being capable of capturing information in challenging visual conditions, event cameras have the potential to overcome the limitations of frame-based cameras in the computer vision and robotics community. In very recent years, deep learning (DL) has been brought to this emerging field and inspired active research endeavors in mining its potential. However, there is still a lack of taxonomies in DL techniques for event-based vision. We first scrutinize the typical event representations with quality enhancement methods as they play a pivotal role as inputs to the DL models. We then provide a comprehensive survey of existing DL-based methods by structurally grouping them into two major categories: 1) image/video reconstruction and restoration; 2) event-based scene understanding and 3D vision. We conduct benchmark experiments for the existing methods in some representative research directions, i.e., image reconstruction, deblurring, and object recognition, to identify some critical insights and problems. Finally, we have discussions regarding the challenges and provide new perspectives for inspiring more research studies.