Differentiable Adaptive 4D Structured Illumination for Joint Capture of Shape and Reflectance

TL;DR

This paper introduces a differentiable framework for adaptively controlling 4D structured illumination to simultaneously capture an object's shape and reflectance with high quality using a single camera.

Contribution

It proposes a novel, unified approach that adaptively adjusts illumination based on pixel-level uncertainty, improving 4D shape and reflectance acquisition.

Findings

Depth results outperform state-of-the-art methods.

Reflectance results are comparable to photographs.

Framework effectively captures diverse shapes and appearances.

Abstract

We present a differentiable framework to adaptively compute 4D illumination conditions with respect to an object, for efficient, high-quality simultaneous acquisition of its shape and reflectance, with a unified spatial-angular structured light and a single camera. Using a simple histogram-based pixel-level probability model for depth and reflectance, we differentiably link the next illumination condition(s) with a loss that encourages the reduction in depth uncertainty. As new structured illumination is cast, corresponding image measurements are used to update the uncertainty at each pixel. Finally, a fine-tuning-based approach reconstructs the depth map and reflectance parameter maps, by minimizing the differences between all physical measurements and their simulated counterparts. The effectiveness of our framework is demonstrated on physical objects with wide variations in shape and appearance. Our depth results compare favorably with state-of-the-art techniques, while our reflectance results are comparable when validated against photographs.