Neural Reflectance for Shape Recovery with Shadow Handling

TL;DR

This paper introduces a self-supervised neural method that accurately recovers complex 3D shapes and reflectance from images with shadows, outperforming prior approaches in accuracy and speed.

Contribution

It presents a novel coordinate-based deep MLP that jointly estimates shape and non-Lambertian reflectance, explicitly predicts shadows, and operates without ground truth data.

Findings

Outperforms existing methods significantly in accuracy.

Achieves an order of magnitude faster processing than CNN-based approaches.

Effectively handles shadows and complex reflectance in real-world images.

Abstract

This paper aims at recovering the shape of a scene with unknown, non-Lambertian, and possibly spatially-varying surface materials. When the shape of the object is highly complex and that shadows cast on the surface, the task becomes very challenging. To overcome these challenges, we propose a coordinate-based deep MLP (multilayer perceptron) to parameterize both the unknown 3D shape and the unknown reflectance at every surface point. This network is able to leverage the observed photometric variance and shadows on the surface, and recover both surface shape and general non-Lambertian reflectance. We explicitly predict cast shadows, mitigating possible artifacts on these shadowing regions, leading to higher estimation accuracy. Our framework is entirely self-supervised, in the sense that it requires neither ground truth shape nor BRDF. Tests on real-world images demonstrate that our method outperform existing methods by a significant margin. Thanks to the small size of the MLP-net, our method is an order of magnitude faster than previous CNN-based methods.