Soft Shadow Diffusion (SSD): Physics-inspired Learning for 3D Computational Periscopy

TL;DR

This paper introduces Soft Shadow Diffusion (SSD), a physics-inspired neural network approach for 3D reconstruction of hidden scenes from ordinary NLOS photographs, overcoming previous resolution and shape limitations.

Contribution

It reformulates the light transport model into a separable inverse problem and develops two solutions, including a novel neural network, for effective 3D NLOS scene reconstruction.

Findings

SSD generalizes well to unseen scenes in simulation and real-world.

The methods are robust to noise and ambient illumination.

Effective 3D scene reconstruction from simple photographs.

Abstract

Conventional imaging requires a line of sight to create accurate visual representations of a scene. In certain circumstances, however, obtaining a suitable line of sight may be impractical, dangerous, or even impossible. Non-line-of-sight (NLOS) imaging addresses this challenge by reconstructing the scene from indirect measurements. Recently, passive NLOS methods that use an ordinary photograph of the subtle shadow cast onto a visible wall by the hidden scene have gained interest. These methods are currently limited to 1D or low-resolution 2D color imaging or to localizing a hidden object whose shape is approximately known. Here, we generalize this class of methods and demonstrate a 3D reconstruction of a hidden scene from an ordinary NLOS photograph. To achieve this, we propose a novel reformulation of the light transport model that conveniently decomposes the hidden scene into \textit{light-occluding} and \textit{non-light-occluding} components to yield a separable non-linear least squares (SNLLS) inverse problem. We develop two solutions: A gradient-based optimization method and a physics-inspired neural network approach, which we call Soft Shadow diffusion (SSD). Despite the challenging ill-conditioned inverse problem encountered here, our approaches are effective on numerous 3D scenes in real experimental scenarios. Moreover, SSD is trained in simulation but generalizes well to unseen classes in simulation and real-world NLOS scenes. SSD also shows surprising robustness to noise and ambient illumination.