Event-based RGB-D sensing with structured light

TL;DR

This paper presents a novel method combining event-based monochrome cameras with structured light projection to achieve RGB-D sensing, enabling depth and color detection of static and moving objects in challenging lighting conditions.

Contribution

It introduces a new approach to detect full RGB events using monochrome event cameras aided by a structured light projector, enabling RGB-D sensing.

Findings

Successfully detects RGB information with monochrome ECs and structured light.

Enables depth and color detection of static and moving objects.

Operates effectively in challenging lighting conditions.

Abstract

Event-based cameras (ECs) are bio-inspired sensors that asynchronously report brightness changes for each pixel. Due to their high dynamic range, pixel bandwidth, temporal resolution, low power consumption, and computational simplicity, they are beneficial for vision-based projects in challenging lighting conditions and they can detect fast movements with their microsecond response time. The first generation of ECs are monochrome, but color data is very useful and sometimes essential for certain vision-based applications. The latest technology enables manufacturers to build color ECs, trading off the size of the sensor and substantially reducing the resolution compared to monochrome models, despite having the same bandwidth. In addition, ECs only detect changes in light and do not show static or slowly moving objects. We introduce a method to detect full RGB events using a monochrome EC aided by a structured light projector. The projector emits rapidly changing RGB patterns of light beams on the scene, the reflection of which is captured by the EC. We combine the benefits of ECs and projection-based techniques and allow depth and color detection of static or moving objects with a commercial TI LightCrafter 4500 projector and a monocular monochrome EC, paving the way for frameless RGB-D sensing applications.