Sharp-It: A Multi-view to Multi-view Diffusion Model for 3D Synthesis and Manipulation

TL;DR

Sharp-It is a novel multi-view diffusion model that enhances low-quality multi-view images to produce high-quality 3D models, combining the strengths of 2D and 3D methods for improved 3D content creation and manipulation.

Contribution

Introduces Sharp-It, a multi-view to multi-view diffusion model that enriches multi-view images for high-quality 3D synthesis and editing, bridging the gap between 2D and 3D approaches.

Findings

Enables high-quality 3D model reconstruction from low-quality multi-view images.

Supports fast synthesis, editing, and controlled generation of 3D assets.

Achieves superior quality compared to traditional multi-view or native 3D generative models.

Abstract

Advancements in text-to-image diffusion models have led to significant progress in fast 3D content creation. One common approach is to generate a set of multi-view images of an object, and then reconstruct it into a 3D model. However, this approach bypasses the use of a native 3D representation of the object and is hence prone to geometric artifacts and limited in controllability and manipulation capabilities. An alternative approach involves native 3D generative models that directly produce 3D representations. These models, however, are typically limited in their resolution, resulting in lower quality 3D objects. In this work, we bridge the quality gap between methods that directly generate 3D representations and ones that reconstruct 3D objects from multi-view images. We introduce a multi-view to multi-view diffusion model called Sharp-It, which takes a 3D consistent set of multi-view images rendered from a low-quality object and enriches its geometric details and texture. The diffusion model operates on the multi-view set in parallel, in the sense that it shares features across the generated views. A high-quality 3D model can then be reconstructed from the enriched multi-view set. By leveraging the advantages of both 2D and 3D approaches, our method offers an efficient and controllable method for high-quality 3D content creation. We demonstrate that Sharp-It enables various 3D applications, such as fast synthesis, editing, and controlled generation, while attaining high-quality assets.