LiDAR-Event Stereo Fusion with Hallucinations

TL;DR

This paper introduces a novel fusion method combining event stereo cameras with LiDAR sensors, using hallucinations to generate fictitious events and improve depth estimation in challenging scenarios.

Contribution

It proposes a new fusion approach that integrates sparse LiDAR depth hints with event stereo data by hallucinating events, enhancing depth estimation accuracy.

Findings

Outperforms state-of-the-art event stereo fusion methods.

Effectively compensates for lack of motion or texture in event data.

Generalizable approach adaptable to various structured representations.

Abstract

Event stereo matching is an emerging technique to estimate depth from neuromorphic cameras; however, events are unlikely to trigger in the absence of motion or the presence of large, untextured regions, making the correspondence problem extremely challenging. Purposely, we propose integrating a stereo event camera with a fixed-frequency active sensor -- e.g., a LiDAR -- collecting sparse depth measurements, overcoming the aforementioned limitations. Such depth hints are used by hallucinating -- i.e., inserting fictitious events -- the stacks or raw input streams, compensating for the lack of information in the absence of brightness changes. Our techniques are general, can be adapted to any structured representation to stack events and outperform state-of-the-art fusion methods applied to event-based stereo.