Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priors

TL;DR

This paper introduces an attentive depth fusion prior for neural implicit representations, improving 3D reconstruction accuracy from multi-view RGBD images by effectively handling occlusions and holes in depth data.

Contribution

It proposes a novel attention mechanism that leverages fused TSDF depth priors, enhancing neural implicit learning from multi-view RGBD images, applicable to static scenes and SLAM.

Findings

Outperforms recent neural implicit methods on benchmark datasets

Effectively handles occlusions and incomplete depth data

Works with both scene-wide and incremental TSDF fusion

Abstract

Learning neural implicit representations has achieved remarkable performance in 3D reconstruction from multi-view images. Current methods use volume rendering to render implicit representations into either RGB or depth images that are supervised by multi-view ground truth. However, rendering a view each time suffers from incomplete depth at holes and unawareness of occluded structures from the depth supervision, which severely affects the accuracy of geometry inference via volume rendering. To resolve this issue, we propose to learn neural implicit representations from multi-view RGBD images through volume rendering with an attentive depth fusion prior. Our prior allows neural networks to perceive coarse 3D structures from the Truncated Signed Distance Function (TSDF) fused from all depth images available for rendering. The TSDF enables accessing the missing depth at holes on one depth image and the occluded parts that are invisible from the current view. By introducing a novel attention mechanism, we allow neural networks to directly use the depth fusion prior with the inferred occupancy as the learned implicit function. Our attention mechanism works with either a one-time fused TSDF that represents a whole scene or an incrementally fused TSDF that represents a partial scene in the context of Simultaneous Localization and Mapping (SLAM). Our evaluations on widely used benchmarks including synthetic and real-world scans show our superiority over the latest neural implicit methods. Project page: https://machineperceptionlab.github.io/Attentive_DF_Prior/