360$^\circ$ Reconstruction From a Single Image Using Space Carved Outpainting

TL;DR

POP3D is a new framework that reconstructs a full 360-degree 3D model from a single image, combining geometric cues, space carving, generative completion, and implicit surface reconstruction to achieve high fidelity and generalizability.

Contribution

It introduces a novel combination of components for single-image 3D reconstruction that generalizes across categories and improves reconstruction quality.

Findings

Outperforms previous methods in reconstruction accuracy.

Achieves high-quality 360-degree models from diverse in-the-wild images.

Demonstrates strong generalization across various object categories.

Abstract

We introduce POP3D, a novel framework that creates a full $360^\circ$-view 3D model from a single image. POP3D resolves two prominent issues that limit the single-view reconstruction. Firstly, POP3D offers substantial generalizability to arbitrary categories, a trait that previous methods struggle to achieve. Secondly, POP3D further improves reconstruction fidelity and naturalness, a crucial aspect that concurrent works fall short of. Our approach marries the strengths of four primary components: (1) a monocular depth and normal predictor that serves to predict crucial geometric cues, (2) a space carving method capable of demarcating the potentially unseen portions of the target object, (3) a generative model pre-trained on a large-scale image dataset that can complete unseen regions of the target, and (4) a neural implicit surface reconstruction method tailored in reconstructing objects using RGB images along with monocular geometric cues. The combination of these components enables POP3D to readily generalize across various in-the-wild images and generate state-of-the-art reconstructions, outperforming similar works by a significant margin. Project page: \url{http://cg.postech.ac.kr/research/POP3D}