Improving Neural Surface Reconstruction with Feature Priors from Multi-View Image

TL;DR

This paper investigates the use of feature priors from various pretext visual tasks to improve neural surface reconstruction, demonstrating that multi-view stereo and patch-wise consistency significantly enhance performance.

Contribution

It explores seven pretext visual tasks and evaluates their effectiveness in strengthening neural surface reconstruction, introducing feature-level consistency and multi-view priors.

Findings

Feature priors from image matching and multi-view stereo outperform others.

Patch-wise feature consistency surpasses pixel-wise methods.

Incorporating pre-trained representations improves reconstruction quality.

Abstract

Recent advancements in Neural Surface Reconstruction (NSR) have significantly improved multi-view reconstruction when coupled with volume rendering. However, relying solely on photometric consistency in image space falls short of addressing complexities posed by real-world data, including occlusions and non-Lambertian surfaces. To tackle these challenges, we propose an investigation into feature-level consistent loss, aiming to harness valuable feature priors from diverse pretext visual tasks and overcome current limitations. It is crucial to note the existing gap in determining the most effective pretext visual task for enhancing NSR. In this study, we comprehensively explore multi-view feature priors from seven pretext visual tasks, comprising thirteen methods. Our main goal is to strengthen NSR training by considering a wide range of possibilities. Additionally, we examine the impact of varying feature resolutions and evaluate both pixel-wise and patch-wise consistent losses, providing insights into effective strategies for improving NSR performance. By incorporating pre-trained representations from MVSFormer and QuadTree, our approach can generate variations of MVS-NeuS and Match-NeuS, respectively. Our results, analyzed on DTU and EPFL datasets, reveal that feature priors from image matching and multi-view stereo outperform other pretext tasks. Moreover, we discover that extending patch-wise photometric consistency to the feature level surpasses the performance of pixel-wise approaches. These findings underscore the effectiveness of these techniques in enhancing NSR outcomes.