Fancy123: One Image to High-Quality 3D Mesh Generation via Plug-and-Play Deformation

TL;DR

Fancy123 introduces a plug-and-play framework with enhancement modules and unprojection to improve the quality, fidelity, and consistency of 3D meshes generated from a single image, outperforming existing methods.

Contribution

It proposes novel enhancement modules and an unprojection operation that significantly improve 3D mesh quality from a single image, with seamless integration into existing methods.

Findings

Achieves state-of-the-art performance in 3D mesh generation from a single image.

Enhancement modules improve multiview consistency and fidelity to input images.

Unprojection discards blurry mesh colors, enhancing clarity.

Abstract

Generating 3D meshes from a single image is an important but ill-posed task. Existing methods mainly adopt 2D multiview diffusion models to generate intermediate multiview images, and use the Large Reconstruction Model (LRM) to create the final meshes. However, the multiview images exhibit local inconsistencies, and the meshes often lack fidelity to the input image or look blurry. We propose Fancy123, featuring two enhancement modules and an unprojection operation to address the above three issues, respectively. The appearance enhancement module deforms the 2D multiview images to realign misaligned pixels for better multiview consistency. The fidelity enhancement module deforms the 3D mesh to match the input image. The unprojection of the input image and deformed multiview images onto LRM's generated mesh ensures high clarity, discarding LRM's predicted blurry-looking mesh colors. Extensive qualitative and quantitative experiments verify Fancy123's SoTA performance with significant improvement. Also, the two enhancement modules are plug-and-play and work at inference time, allowing seamless integration into various existing single-image-to-3D methods. Code at: https://github.com/YuQiao0303/Fancy123