Neural Signed Distance Function Inference through Splatting 3D Gaussians Pulled on Zero-Level Set

TL;DR

This paper introduces a novel method that combines 3D Gaussian splatting with neural SDF learning, using a differentiable pulling technique to improve surface reconstruction accuracy and detail in multi-view settings.

Contribution

It proposes a dynamic alignment and joint optimization approach that effectively constrains neural SDF inference with multi-view consistency and 3D Gaussian splatting.

Findings

Achieves more accurate and smooth surface reconstructions.

Outperforms state-of-the-art methods on benchmark datasets.

Enhances geometry detail and surface completeness.

Abstract

It is vital to infer a signed distance function (SDF) in multi-view based surface reconstruction. 3D Gaussian splatting (3DGS) provides a novel perspective for volume rendering, and shows advantages in rendering efficiency and quality. Although 3DGS provides a promising neural rendering option, it is still hard to infer SDFs for surface reconstruction with 3DGS due to the discreteness, the sparseness, and the off-surface drift of 3D Gaussians. To resolve these issues, we propose a method that seamlessly merge 3DGS with the learning of neural SDFs. Our key idea is to more effectively constrain the SDF inference with the multi-view consistency. To this end, we dynamically align 3D Gaussians on the zero-level set of the neural SDF using neural pulling, and then render the aligned 3D Gaussians through the differentiable rasterization. Meanwhile, we update the neural SDF by pulling neighboring space to the pulled 3D Gaussians, which progressively refine the signed distance field near the surface. With both differentiable pulling and splatting, we jointly optimize 3D Gaussians and the neural SDF with both RGB and geometry constraints, which recovers more accurate, smooth, and complete surfaces with more geometry details. Our numerical and visual comparisons show our superiority over the state-of-the-art results on the widely used benchmarks.