Pseudo-labels for Supervised Learning on Dynamic Vision Sensor Data, Applied to Object Detection under Ego-motion

TL;DR

This paper introduces a method to generate pseudo-labels for training event-based neural networks, enabling real-time object detection under ego-motion with promising accuracy, and demonstrating the potential of event-based sensors in challenging conditions.

Contribution

It presents the first event-based car detection system under ego-motion using pseudo-labels transferred from frame-based CNNs, advancing event-based vision applications.

Findings

Achieved 40.3% average precision in real-world ego-motion scenarios.

Event-based detector operates at 100 fps with high temporal resolution.

Complemented frame-based detectors, indicating learned generalized visual features.

Abstract

In recent years, dynamic vision sensors (DVS), also known as event-based cameras or neuromorphic sensors, have seen increased use due to various advantages over conventional frame-based cameras. Using principles inspired by the retina, its high temporal resolution overcomes motion blurring, its high dynamic range overcomes extreme illumination conditions and its low power consumption makes it ideal for embedded systems on platforms such as drones and self-driving cars. However, event-based data sets are scarce and labels are even rarer for tasks such as object detection. We transferred discriminative knowledge from a state-of-the-art frame-based convolutional neural network (CNN) to the event-based modality via intermediate pseudo-labels, which are used as targets for supervised learning. We show, for the first time, event-based car detection under ego-motion in a real environment at 100 frames per second with a test average precision of 40.3% relative to our annotated ground truth. The event-based car detector handles motion blur and poor illumination conditions despite not explicitly trained to do so, and even complements frame-based CNN detectors, suggesting that it has learnt generalized visual representations.