PVSeRF: Joint Pixel-, Voxel- and Surface-Aligned Radiance Field for Single-Image Novel View Synthesis

TL;DR

PVSeRF is a novel framework that enhances single-image 3D reconstruction and view synthesis by integrating pixel-, voxel-, and surface-aligned features, improving geometry accuracy and image quality.

Contribution

It introduces a geometry-aware radiance field model combining pixel-, voxel-, and surface-aligned features for better disentanglement of appearance and geometry.

Findings

Outperforms state-of-the-art methods on ShapeNet benchmarks.

Produces more accurate geometries and higher quality novel view images.

Effectively disentangles appearance and geometry in single-image reconstruction.

Abstract

We present PVSeRF, a learning framework that reconstructs neural radiance fields from single-view RGB images, for novel view synthesis. Previous solutions, such as pixelNeRF, rely only on pixel-aligned features and suffer from feature ambiguity issues. As a result, they struggle with the disentanglement of geometry and appearance, leading to implausible geometries and blurry results. To address this challenge, we propose to incorporate explicit geometry reasoning and combine it with pixel-aligned features for radiance field prediction. Specifically, in addition to pixel-aligned features, we further constrain the radiance field learning to be conditioned on i) voxel-aligned features learned from a coarse volumetric grid and ii) fine surface-aligned features extracted from a regressed point cloud. We show that the introduction of such geometry-aware features helps to achieve a better disentanglement between appearance and geometry, i.e. recovering more accurate geometries and synthesizing higher quality images of novel views. Extensive experiments against state-of-the-art methods on ShapeNet benchmarks demonstrate the superiority of our approach for single-image novel view synthesis.