RayDF: Neural Ray-surface Distance Fields with Multi-view Consistency

TL;DR

RayDF introduces a novel neural framework for 3D shape representation that ensures multi-view consistency, enabling faster rendering and improved surface reconstruction compared to existing methods.

Contribution

The paper presents RayDF, a new ray-based neural shape representation with multi-view consistency, dual-ray visibility classifier, and significant speed and accuracy improvements.

Findings

Achieves superior 3D surface point reconstruction on multiple datasets.

Surpasses existing coordinate-based and ray-based methods in performance.

Enables 1000x faster rendering of depth images.

Abstract

In this paper, we study the problem of continuous 3D shape representations. The majority of existing successful methods are coordinate-based implicit neural representations. However, they are inefficient to render novel views or recover explicit surface points. A few works start to formulate 3D shapes as ray-based neural functions, but the learned structures are inferior due to the lack of multi-view geometry consistency. To tackle these challenges, we propose a new framework called RayDF. It consists of three major components: 1) the simple ray-surface distance field, 2) the novel dual-ray visibility classifier, and 3) a multi-view consistency optimization module to drive the learned ray-surface distances to be multi-view geometry consistent. We extensively evaluate our method on three public datasets, demonstrating remarkable performance in 3D surface point reconstruction on both synthetic and challenging real-world 3D scenes, clearly surpassing existing coordinate-based and ray-based baselines. Most notably, our method achieves a 1000x faster speed than coordinate-based methods to render an 800x800 depth image, showing the superiority of our method for 3D shape representation. Our code and data are available at https://github.com/vLAR-group/RayDF