Incorporating dense metric depth into neural 3D representations for view synthesis and relighting

TL;DR

This paper introduces a method that integrates dense metric depth into neural 3D representations to enhance view synthesis and relighting, especially useful in robotics where limited viewpoints and scene clutter pose challenges.

Contribution

It presents a novel approach to incorporate dense depth data into neural 3D models, improving geometry and appearance estimation in scenes with limited viewpoints.

Findings

Improved quality of scene geometry and appearance estimation.

Effective relighting and view synthesis with few training views.

Addressed artifacts in joint geometry and appearance refinement.

Abstract

Synthesizing accurate geometry and photo-realistic appearance of small scenes is an active area of research with compelling use cases in gaming, virtual reality, robotic-manipulation, autonomous driving, convenient product capture, and consumer-level photography. When applying scene geometry and appearance estimation techniques to robotics, we found that the narrow cone of possible viewpoints due to the limited range of robot motion and scene clutter caused current estimation techniques to produce poor quality estimates or even fail. On the other hand, in robotic applications, dense metric depth can often be measured directly using stereo and illumination can be controlled. Depth can provide a good initial estimate of the object geometry to improve reconstruction, while multi-illumination images can facilitate relighting. In this work we demonstrate a method to incorporate dense metric depth into the training of neural 3D representations and address an artifact observed while jointly refining geometry and appearance by disambiguating between texture and geometry edges. We also discuss a multi-flash stereo camera system developed to capture the necessary data for our pipeline and show results on relighting and view synthesis with a few training views.