E-RGB-D: Real-Time Event-Based Perception with Structured Light

TL;DR

This paper introduces a real-time RGB-D perception system combining event-based cameras with structured light projection, enabling fast, color, and depth sensing for dynamic scenes with high resolution and bandwidth efficiency.

Contribution

The novel integration of event-based cameras with active structured light using a DLP projector for simultaneous color and depth sensing in real-time.

Findings

Color detection speed of 1400 fps

Pixel depth detection at 4 kHz

Enables frameless RGB-D sensing applications

Abstract

Event-based cameras (ECs) have emerged as bio-inspired sensors that report pixel brightness changes asynchronously, offering unmatched speed and efficiency in vision sensing. Despite their high dynamic range, temporal resolution, low power consumption, and computational simplicity, traditional monochrome ECs face limitations in detecting static or slowly moving objects and lack color information essential for certain applications. To address these challenges, we present a novel approach that integrates a Digital Light Processing (DLP) projector, forming Active Structured Light (ASL) for RGB-D sensing. By combining the benefits of ECs and projection-based techniques, our method enables the detection of color and the depth of each pixel separately. Dynamic projection adjustments optimize bandwidth, ensuring selective color data acquisition and yielding colorful point clouds without sacrificing spatial resolution. This integration, facilitated by a commercial TI LightCrafter 4500 projector and a monocular monochrome EC, not only enables frameless RGB-D sensing applications but also achieves remarkable performance milestones. With our approach, we achieved a color detection speed equivalent to 1400 fps and 4 kHz of pixel depth detection, significantly advancing the realm of computer vision across diverse fields from robotics to 3D reconstruction methods. Our code is publicly available: https://github.com/MISTLab/event_based_rgbd_ros