Robust Single-shot Structured Light 3D Imaging via Neural Feature Decoding

TL;DR

This paper introduces a neural feature decoding framework for single-shot structured light 3D imaging that significantly improves robustness and detail recovery by matching features in a learned space rather than pixel domain, trained on synthetic data.

Contribution

The authors propose a novel learning-based structured light decoding method using neural features and a depth refinement module, trained on synthetic data, that outperforms traditional pixel-based methods and generalizes well to real environments.

Findings

Substantial performance improvements over pixel-domain decoding.

Effective generalization to real-world indoor environments.

Outperforms commercial structured light and passive stereo methods.

Abstract

We consider the problem of active 3D imaging using single-shot structured light systems, which are widely employed in commercial 3D sensing devices such as Apple Face ID and Intel RealSense. Traditional structured light methods typically decode depth correspondences through pixel-domain matching algorithms, resulting in limited robustness under challenging scenarios like occlusions, fine-structured details, and non-Lambertian surfaces. Inspired by recent advances in neural feature matching, we propose a learning-based structured light decoding framework that performs robust correspondence matching within feature space rather than the fragile pixel domain. Our method extracts neural features from the projected patterns and captured infrared (IR) images, explicitly incorporating their geometric priors by building cost volumes in feature space, achieving substantial performance improvements over pixel-domain decoding approaches. To further enhance depth quality, we introduce a depth refinement module that leverages strong priors from large-scale monocular depth estimation models, improving fine detail recovery and global structural coherence. To facilitate effective learning, we develop a physically-based structured light rendering pipeline, generating nearly one million synthetic pattern-image pairs with diverse objects and materials for indoor settings. Experiments demonstrate that our method, trained exclusively on synthetic data with multiple structured light patterns, generalizes well to real-world indoor environments, effectively processes various pattern types without retraining, and consistently outperforms both commercial structured light systems and passive stereo RGB-based depth estimation methods. Project page: https://namisntimpot.github.io/NSLweb/.