Cross-spectral Gated-RGB Stereo Depth Estimation

TL;DR

This paper introduces a novel multi-modal stereo depth estimation system combining gated, RGB, and NIR imaging with a new algorithm, achieving significantly improved long-range depth accuracy over existing methods.

Contribution

It presents a new multi-view, multi-modal stereo depth estimation method leveraging gated and RGB imaging, outperforming existing techniques at long ranges.

Findings

Achieves 39% lower MAE in depth estimation at 100-220m range.

Utilizes low-cost CMOS sensors and flood illumination for high-resolution depth capture.

Outperforms previous methods in long-range depth accuracy.

Abstract

Gated cameras flood-illuminate a scene and capture the time-gated impulse response of a scene. By employing nanosecond-scale gates, existing sensors are capable of capturing mega-pixel gated images, delivering dense depth improving on today's LiDAR sensors in spatial resolution and depth precision. Although gated depth estimation methods deliver a million of depth estimates per frame, their resolution is still an order below existing RGB imaging methods. In this work, we combine high-resolution stereo HDR RCCB cameras with gated imaging, allowing us to exploit depth cues from active gating, multi-view RGB and multi-view NIR sensing -- multi-view and gated cues across the entire spectrum. The resulting capture system consists only of low-cost CMOS sensors and flood-illumination. We propose a novel stereo-depth estimation method that is capable of exploiting these multi-modal multi-view depth cues, including the active illumination that is measured by the RCCB camera when removing the IR-cut filter. The proposed method achieves accurate depth at long ranges, outperforming the next best existing method by 39% for ranges of 100 to 220m in MAE on accumulated LiDAR ground-truth. Our code, models and datasets are available at https://light.princeton.edu/gatedrccbstereo/ .