NEAT: Distilling 3D Wireframes from Neural Attraction Fields

TL;DR

This paper introduces NEAT, a neural field-based method for 3D wireframe reconstruction from 2D observations, which jointly optimizes neural representations and junctions without precomputed matching, outperforming existing methods.

Contribution

NEAT presents a novel rendering-distilling approach that jointly optimizes neural fields and global junctions for 3D wireframe reconstruction from scratch using view-dependent 2D data.

Findings

Outperforms state-of-the-art 3D wireframe reconstruction methods on DTU and BlendedMVS datasets.

Produces 3D global junctions that serve as superior initialization for 3D Gaussian Splatting.

Requires about 20 times fewer initial 3D points for high-fidelity view synthesis.

Abstract

This paper studies the problem of structured 3D reconstruction using wireframes that consist of line segments and junctions, focusing on the computation of structured boundary geometries of scenes. Instead of leveraging matching-based solutions from 2D wireframes (or line segments) for 3D wireframe reconstruction as done in prior arts, we present NEAT, a rendering-distilling formulation using neural fields to represent 3D line segments with 2D observations, and bipartite matching for perceiving and distilling of a sparse set of 3D global junctions. The proposed {NEAT} enjoys the joint optimization of the neural fields and the global junctions from scratch, using view-dependent 2D observations without precomputed cross-view feature matching. Comprehensive experiments on the DTU and BlendedMVS datasets demonstrate our NEAT's superiority over state-of-the-art alternatives for 3D wireframe reconstruction. Moreover, the distilled 3D global junctions by NEAT, are a better initialization than SfM points, for the recently-emerged 3D Gaussian Splatting for high-fidelity novel view synthesis using about 20 times fewer initial 3D points. Project page: \url{https://xuenan.net/neat}.