Neural RGB-D Surface Reconstruction

TL;DR

This paper introduces a novel neural surface reconstruction method that integrates depth data into a NeRF framework using an implicit surface representation, enabling high-quality, metric 3D reconstructions of room-scale scenes.

Contribution

It presents a new approach combining implicit surface representation with NeRFs and depth data, improving 3D reconstruction quality over existing volumetric methods.

Findings

Achieves high-quality, metric 3D reconstructions.

Incorporates depth measurements from RGB-D sensors.

Includes pose and camera refinement techniques.

Abstract

Obtaining high-quality 3D reconstructions of room-scale scenes is of paramount importance for upcoming applications in AR or VR. These range from mixed reality applications for teleconferencing, virtual measuring, virtual room planing, to robotic applications. While current volume-based view synthesis methods that use neural radiance fields (NeRFs) show promising results in reproducing the appearance of an object or scene, they do not reconstruct an actual surface. The volumetric representation of the surface based on densities leads to artifacts when a surface is extracted using Marching Cubes, since during optimization, densities are accumulated along the ray and are not used at a single sample point in isolation. Instead of this volumetric representation of the surface, we propose to represent the surface using an implicit function (truncated signed distance function). We show how to incorporate this representation in the NeRF framework, and extend it to use depth measurements from a commodity RGB-D sensor, such as a Kinect. In addition, we propose a pose and camera refinement technique which improves the overall reconstruction quality. In contrast to concurrent work on integrating depth priors in NeRF which concentrates on novel view synthesis, our approach is able to reconstruct high-quality, metrical 3D reconstructions.