Active Stereo in the Wild through Virtual Pattern Projection

TL;DR

This paper introduces a virtual pattern projection method that replaces physical projectors with depth sensors, enabling robust and accurate stereo matching in diverse environments without hardware modifications.

Contribution

It proposes a flexible, general-purpose guided stereo approach using virtual patterns from depth sensors, overcoming physical projector limitations and achieving state-of-the-art results.

Findings

Boosts robustness and accuracy of stereo algorithms

Works with off-the-shelf depth sensors in various environments

Achieves state-of-the-art performance on multiple datasets

Abstract

This paper presents a novel general-purpose guided stereo paradigm that mimics the active stereo principle by replacing the unreliable physical pattern projector with a depth sensor. It works by projecting virtual patterns consistent with the scene geometry onto the left and right images acquired by a conventional stereo camera, using the sparse hints obtained from a depth sensor, to facilitate the visual correspondence. Purposely, any depth sensing device can be seamlessly plugged into our framework, enabling the deployment of a virtual active stereo setup in any possible environment and overcoming the severe limitations of physical pattern projection, such as the limited working range and environmental conditions. Exhaustive experiments on indoor and outdoor datasets featuring both long and close range, including those providing raw, unfiltered depth hints from off-the-shelf depth sensors, highlight the effectiveness of our approach in notably boosting the robustness and accuracy of algorithms and deep stereo without any code modification and even without re-training. Additionally, we assess the performance of our strategy on active stereo evaluation datasets with conventional pattern projection. Indeed, in all these scenarios, our virtual pattern projection paradigm achieves state-of-the-art performance. The source code is available at: https://github.com/bartn8/vppstereo.