ConRad: Image Constrained Radiance Fields for 3D Generation from a Single Image

TL;DR

ConRad introduces a new neural radiance field variant that reconstructs 3D objects from a single RGB image, effectively capturing appearance and details while maintaining fidelity to the input.

Contribution

The paper presents ConRad, a novel 3D representation that explicitly encodes input image appearance and a training method leveraging pretrained diffusion models for single-image 3D reconstruction.

Findings

ConRad produces more faithful 3D reconstructions compared to baselines.

It preserves image details and appearance more effectively.

Achieves superior quantitative performance on ShapeNet benchmark.

Abstract

We present a novel method for reconstructing 3D objects from a single RGB image. Our method leverages the latest image generation models to infer the hidden 3D structure while remaining faithful to the input image. While existing methods obtain impressive results in generating 3D models from text prompts, they do not provide an easy approach for conditioning on input RGB data. Na\"ive extensions of these methods often lead to improper alignment in appearance between the input image and the 3D reconstructions. We address these challenges by introducing Image Constrained Radiance Fields (ConRad), a novel variant of neural radiance fields. ConRad is an efficient 3D representation that explicitly captures the appearance of an input image in one viewpoint. We propose a training algorithm that leverages the single RGB image in conjunction with pretrained Diffusion Models to optimize the parameters of a ConRad representation. Extensive experiments show that ConRad representations can simplify preservation of image details while producing a realistic 3D reconstruction. Compared to existing state-of-the-art baselines, we show that our 3D reconstructions remain more faithful to the input and produce more consistent 3D models while demonstrating significantly improved quantitative performance on a ShapeNet object benchmark.