MVBoost: Boost 3D Reconstruction with Multi-View Refinement

TL;DR

MVBoost introduces a multi-view refinement framework that generates pseudo-ground-truth data from single images, significantly improving 3D reconstruction accuracy and generalization by combining multi-view diffusion and 3D modeling.

Contribution

The paper presents a novel framework that leverages multi-view diffusion and 3D reconstruction to create large-scale training data, enhancing 3D reconstruction performance and generalization.

Findings

Achieves superior 3D reconstruction accuracy

Demonstrates robust generalization across datasets

Provides effective multi-view data generation method

Abstract

Recent advancements in 3D object reconstruction have been remarkable, yet most current 3D models rely heavily on existing 3D datasets. The scarcity of diverse 3D datasets results in limited generalization capabilities of 3D reconstruction models. In this paper, we propose a novel framework for boosting 3D reconstruction with multi-view refinement (MVBoost) by generating pseudo-GT data. The key of MVBoost is combining the advantages of the high accuracy of the multi-view generation model and the consistency of the 3D reconstruction model to create a reliable data source. Specifically, given a single-view input image, we employ a multi-view diffusion model to generate multiple views, followed by a large 3D reconstruction model to produce consistent 3D data. MVBoost then adaptively refines these multi-view images, rendered from the consistent 3D data, to build a large-scale multi-view dataset for training a feed-forward 3D reconstruction model. Additionally, the input view optimization is designed to optimize the corresponding viewpoints based on the user's input image, ensuring that the most important viewpoint is accurately tailored to the user's needs. Extensive evaluations demonstrate that our method achieves superior reconstruction results and robust generalization compared to prior works.