EventHub: Data Factory for Generalizable Event-Based Stereo Networks without Active Sensors

TL;DR

EventHub introduces a new framework for training event-based stereo networks using standard images and view synthesis, eliminating the need for costly active sensors and ground truth annotations.

Contribution

It presents a novel data factory that generates proxy annotations and events from images, enabling the adaptation of RGB stereo models to event data with improved generalization.

Findings

EventHub achieves state-of-the-art results on event stereo datasets.

The framework improves RGB stereo models' performance in nighttime conditions.

Proxy data generation enhances event stereo network training without ground truth.

Abstract

We propose EventHub, a novel framework for training deep-event stereo networks without ground truth annotations from costly active sensors, relying instead on standard color images. From these images, we derive either proxy annotations and proxy events through state-of-the-art novel view synthesis techniques, or simply proxy annotations when images are already paired with event data. Using the training set generated by our data factory, we repurpose state-of-the-art stereo models from RGB literature to process event data, obtaining new event stereo models with unprecedented generalization capabilities. Experiments on widely used event stereo datasets support the effectiveness of EventHub and show how the same data distillation mechanism can improve the accuracy of RGB stereo foundation models in challenging conditions such as nighttime scenes.