Stereo Hybrid Event-Frame (SHEF) Cameras for 3D Perception

TL;DR

This paper introduces a novel stereo hybrid event-frame (SHEF) camera system that combines pure event and frame sensors to improve 3D perception in robotics, along with a new dataset and disparity estimation algorithm.

Contribution

It presents the SHEF camera system with separate high-quality event and frame sensors, and a disparity estimation algorithm that outperforms existing methods.

Findings

Disparity estimation algorithm surpasses state-of-the-art performance.

SHEF dataset enables effective evaluation of stereo depth algorithms.

Hybrid sensor approach overcomes limitations of pure event or frame cameras.

Abstract

Stereo camera systems play an important role in robotics applications to perceive the 3D world. However, conventional cameras have drawbacks such as low dynamic range, motion blur and latency due to the underlying frame-based mechanism. Event cameras address these limitations as they report the brightness changes of each pixel independently with a fine temporal resolution, but they are unable to acquire absolute intensity information directly. Although integrated hybrid event-frame sensors (eg., DAVIS) are available, the quality of data is compromised by coupling at the pixel level in the circuit fabrication of such cameras. This paper proposes a stereo hybrid event-frame (SHEF) camera system that offers a sensor modality with separate high-quality pure event and pure frame cameras, overcoming the limitations of each separate sensor and allowing for stereo depth estimation. We provide a SHEF dataset targeted at evaluating disparity estimation algorithms and introduce a stereo disparity estimation algorithm that uses edge information extracted from the event stream correlated with the edge detected in the frame data. Our disparity estimation outperforms the state-of-the-art stereo matching algorithm on the SHEF dataset.