GaussianObject: High-Quality 3D Object Reconstruction from Four Views with Gaussian Splatting

TL;DR

GaussianObject is a novel framework that reconstructs high-quality 3D objects from only four sparse views using Gaussian splatting, structure priors, and diffusion-based refinement, outperforming previous methods.

Contribution

The paper introduces GaussianObject, a new approach combining structure priors and diffusion models for 3D reconstruction from minimal views, with a COLMAP-free variant for broader applicability.

Findings

Achieves superior 3D reconstruction quality from four views.

Outperforms previous state-of-the-art methods on multiple datasets.

Provides a COLMAP-free variant for unposed images.

Abstract

Reconstructing and rendering 3D objects from highly sparse views is of critical importance for promoting applications of 3D vision techniques and improving user experience. However, images from sparse views only contain very limited 3D information, leading to two significant challenges: 1) Difficulty in building multi-view consistency as images for matching are too few; 2) Partially omitted or highly compressed object information as view coverage is insufficient. To tackle these challenges, we propose GaussianObject, a framework to represent and render the 3D object with Gaussian splatting that achieves high rendering quality with only 4 input images. We first introduce techniques of visual hull and floater elimination, which explicitly inject structure priors into the initial optimization process to help build multi-view consistency, yielding a coarse 3D Gaussian representation. Then we construct a Gaussian repair model based on diffusion models to supplement the omitted object information, where Gaussians are further refined. We design a self-generating strategy to obtain image pairs for training the repair model. We further design a COLMAP-free variant, where pre-given accurate camera poses are not required, which achieves competitive quality and facilitates wider applications. GaussianObject is evaluated on several challenging datasets, including MipNeRF360, OmniObject3D, OpenIllumination, and our-collected unposed images, achieving superior performance from only four views and significantly outperforming previous SOTA methods. Our demo is available at https://gaussianobject.github.io/, and the code has been released at https://github.com/GaussianObject/GaussianObject.