ShaRF: Shape-conditioned Radiance Fields from a Single View

TL;DR

ShaRF introduces a novel method for reconstructing and rendering 3D objects from a single image by estimating a geometric scaffold and disentangling shape and appearance, enabling realistic view synthesis and generalization.

Contribution

The paper presents a shape-conditioned radiance field approach that estimates a geometric scaffold from a single image, allowing for faithful 3D reconstruction and view synthesis with disentangled shape and appearance.

Findings

Effective single-view 3D reconstruction demonstrated on synthetic and real images.

Model generalizes to out-of-domain images, including photographs.

Accurate geometric scaffold as a 3D shape estimate.

Abstract

We present a method for estimating neural scenes representations of objects given only a single image. The core of our method is the estimation of a geometric scaffold for the object and its use as a guide for the reconstruction of the underlying radiance field. Our formulation is based on a generative process that first maps a latent code to a voxelized shape, and then renders it to an image, with the object appearance being controlled by a second latent code. During inference, we optimize both the latent codes and the networks to fit a test image of a new object. The explicit disentanglement of shape and appearance allows our model to be fine-tuned given a single image. We can then render new views in a geometrically consistent manner and they represent faithfully the input object. Additionally, our method is able to generalize to images outside of the training domain (more realistic renderings and even real photographs). Finally, the inferred geometric scaffold is itself an accurate estimate of the object's 3D shape. We demonstrate in several experiments the effectiveness of our approach in both synthetic and real images.