SERES: Semantic-aware neural reconstruction from sparse views

TL;DR

This paper introduces SERES, a semantic-aware neural reconstruction method that enhances 3D model quality from sparse images by integrating semantic logits and geometric regularization, significantly reducing reconstruction errors.

Contribution

The paper presents a novel semantic-aware neural reconstruction approach that incorporates patch-based semantic logits and geometric primitive masks to improve accuracy from sparse views.

Findings

Reduces average chamfer distance by 44% on SparseNeuS and 20% on VolRecon.

Decreases error by 69% when integrated with NeuS as a plugin.

Enhances reconstruction quality significantly in sparse view scenarios.

Abstract

We propose a semantic-aware neural reconstruction method to generate 3D high-fidelity models from sparse images. To tackle the challenge of severe radiance ambiguity caused by mismatched features in sparse input, we enrich neural implicit representations by adding patch-based semantic logits that are optimized together with the signed distance field and the radiance field. A novel regularization based on the geometric primitive masks is introduced to mitigate shape ambiguity. The performance of our approach has been verified in experimental evaluation. The average chamfer distances of our reconstruction on the DTU dataset can be reduced by 44% for SparseNeuS and 20% for VolRecon. When working as a plugin for those dense reconstruction baselines such as NeuS and Neuralangelo, the average error on the DTU dataset can be reduced by 69% and 68% respectively.