Towards Learning Neural Representations from Shadows

TL;DR

This paper introduces a neural shadow field method that learns scene geometry solely from shadows, outperforming traditional shape-from-shadow and integrating with neural rendering for improved 3D reconstruction.

Contribution

It presents a novel differentiable volumetric rendering approach that leverages shadows as a key cue for scene reconstruction, enhancing generalization and accuracy.

Findings

Shadows provide strong cues for geometry estimation.

Neural shadow fields outperform traditional shape-from-shadow methods.

The approach integrates with existing neural rendering techniques.

Abstract

We present a method that learns neural shadow fields which are neural scene representations that are only learnt from the shadows present in the scene. While traditional shape-from-shadow (SfS) algorithms reconstruct geometry from shadows, they assume a fixed scanning setup and fail to generalize to complex scenes. Neural rendering algorithms, on the other hand, rely on photometric consistency between RGB images, but largely ignore physical cues such as shadows, which have been shown to provide valuable information about the scene. We observe that shadows are a powerful cue that can constrain neural scene representations to learn SfS, and even outperform NeRF to reconstruct otherwise hidden geometry. We propose a graphics-inspired differentiable approach to render accurate shadows with volumetric rendering, predicting a shadow map that can be compared to the ground truth shadow. Even with just binary shadow maps, we show that neural rendering can localize the object and estimate coarse geometry. Our approach reveals that sparse cues in images can be used to estimate geometry using differentiable volumetric rendering. Moreover, our framework is highly generalizable and can work alongside existing 3D reconstruction techniques that otherwise only use photometric consistency.