Deep Reflectance Volumes: Relightable Reconstructions from Multi-View Photometric Images

TL;DR

This paper introduces Deep Reflectance Volumes, a deep learning method for reconstructing and relighting scenes from multi-view images with complex reflectance, geometry, and lighting, enabling photorealistic rendering and material editing.

Contribution

It proposes a novel volumetric scene representation and a differentiable volume ray marching framework for relightable scene reconstruction from unstructured images.

Findings

Reconstructs scenes with non-Lambertian reflectance and complex geometry.

Achieves photorealistic rendering under novel viewpoints and lighting.

Outperforms state-of-the-art mesh-based methods.

Abstract

We present a deep learning approach to reconstruct scene appearance from unstructured images captured under collocated point lighting. At the heart of Deep Reflectance Volumes is a novel volumetric scene representation consisting of opacity, surface normal and reflectance voxel grids. We present a novel physically-based differentiable volume ray marching framework to render these scene volumes under arbitrary viewpoint and lighting. This allows us to optimize the scene volumes to minimize the error between their rendered images and the captured images. Our method is able to reconstruct real scenes with challenging non-Lambertian reflectance and complex geometry with occlusions and shadowing. Moreover, it accurately generalizes to novel viewpoints and lighting, including non-collocated lighting, rendering photorealistic images that are significantly better than state-of-the-art mesh-based methods. We also show that our learned reflectance volumes are editable, allowing for modifying the materials of the captured scenes.