Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation

TL;DR

This paper introduces an unsupervised domain adaptation method that transfers models trained on labeled images to unlabeled event data, enhancing event-based vision tasks without relying on paired sensor data.

Contribution

It proposes a novel task transfer approach that leverages a generative event model to split features, enabling effective domain adaptation from images to event data without paired datasets.

Findings

Outperforms existing methods in object detection by 0.26 mAP (93% increase)

Improves classification accuracy by 2.7%

Enables training of event-based neural networks using existing image datasets

Abstract

Reliable perception during fast motion maneuvers or in high dynamic range environments is crucial for robotic systems. Since event cameras are robust to these challenging conditions, they have great potential to increase the reliability of robot vision. However, event-based vision has been held back by the shortage of labeled datasets due to the novelty of event cameras. To overcome this drawback, we propose a task transfer method to train models directly with labeled images and unlabeled event data. Compared to previous approaches, (i) our method transfers from single images to events instead of high frame rate videos, and (ii) does not rely on paired sensor data. To achieve this, we leverage the generative event model to split event features into content and motion features. This split enables efficient matching between latent spaces for events and images, which is crucial for successful task transfer. Thus, our approach unlocks the vast amount of existing image datasets for the training of event-based neural networks. Our task transfer method consistently outperforms methods targeting Unsupervised Domain Adaptation for object detection by 0.26 mAP (increase by 93%) and classification by 2.7% accuracy.