GenS: Generalizable Neural Surface Reconstruction from Multi-View Images

TL;DR

GenS is a neural surface reconstruction model that generalizes across scenes, using a multi-scale volume and feature-metric consistency to recover detailed surfaces from multi-view images without scene-specific training.

Contribution

It introduces a generalized multi-scale volume representation and multi-scale feature-metric consistency for scene-agnostic neural surface reconstruction.

Findings

Outperforms state-of-the-art methods on benchmarks.

Effectively generalizes to new scenes without per-scene optimization.

Recovers high-frequency details while maintaining global smoothness.

Abstract

Combining the signed distance function (SDF) and differentiable volume rendering has emerged as a powerful paradigm for surface reconstruction from multi-view images without 3D supervision. However, current methods are impeded by requiring long-time per-scene optimizations and cannot generalize to new scenes. In this paper, we present GenS, an end-to-end generalizable neural surface reconstruction model. Unlike coordinate-based methods that train a separate network for each scene, we construct a generalized multi-scale volume to directly encode all scenes. Compared with existing solutions, our representation is more powerful, which can recover high-frequency details while maintaining global smoothness. Meanwhile, we introduce a multi-scale feature-metric consistency to impose the multi-view consistency in a more discriminative multi-scale feature space, which is robust to the failures of the photometric consistency. And the learnable feature can be self-enhanced to continuously improve the matching accuracy and mitigate aggregation ambiguity. Furthermore, we design a view contrast loss to force the model to be robust to those regions covered by few viewpoints through distilling the geometric prior from dense input to sparse input. Extensive experiments on popular benchmarks show that our model can generalize well to new scenes and outperform existing state-of-the-art methods even those employing ground-truth depth supervision. Code is available at https://github.com/prstrive/GenS.