Improved Neural Radiance Fields Using Pseudo-depth and Fusion

TL;DR

This paper introduces a multi-scale NeRF approach that integrates pseudo-depth prediction and feature fusion to improve novel view synthesis and geometry modeling without per-scene optimization.

Contribution

It proposes constructing multi-scale encoding volumes and depth-guided feature fusion to enhance geometric accuracy and generalization in NeRF models.

Findings

Superior performance in novel view synthesis

Improved dense geometry modeling

Effective without per-scene optimization

Abstract

Since the advent of Neural Radiance Fields, novel view synthesis has received tremendous attention. The existing approach for the generalization of radiance field reconstruction primarily constructs an encoding volume from nearby source images as additional inputs. However, these approaches cannot efficiently encode the geometric information of real scenes with various scale objects/structures. In this work, we propose constructing multi-scale encoding volumes and providing multi-scale geometry information to NeRF models. To make the constructed volumes as close as possible to the surfaces of objects in the scene and the rendered depth more accurate, we propose to perform depth prediction and radiance field reconstruction simultaneously. The predicted depth map will be used to supervise the rendered depth, narrow the depth range, and guide points sampling. Finally, the geometric information contained in point volume features may be inaccurate due to occlusion, lighting, etc. To this end, we propose enhancing the point volume feature from depth-guided neighbor feature fusion. Experiments demonstrate the superior performance of our method in both novel view synthesis and dense geometry modeling without per-scene optimization.