3D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation

TL;DR

3D-Adapter is a plug-in module that enhances pretrained image diffusion models with 3D geometry awareness, significantly improving 3D generation quality and consistency across various tasks.

Contribution

The paper introduces 3D-Adapter, a novel plug-in that incorporates 3D feedback into diffusion models, enabling geometry-consistent 3D generation from text and images.

Findings

Enhances geometry quality of multi-view diffusion models.

Enables high-quality 3D generation with plain text-to-image models.

Supports diverse applications like text-to-3D and image-to-3D.

Abstract

Multi-view image diffusion models have significantly advanced open-domain 3D object generation. However, most existing models rely on 2D network architectures that lack inherent 3D biases, resulting in compromised geometric consistency. To address this challenge, we introduce 3D-Adapter, a plug-in module designed to infuse 3D geometry awareness into pretrained image diffusion models. Central to our approach is the idea of 3D feedback augmentation: for each denoising step in the sampling loop, 3D-Adapter decodes intermediate multi-view features into a coherent 3D representation, then re-encodes the rendered RGBD views to augment the pretrained base model through feature addition. We study two variants of 3D-Adapter: a fast feed-forward version based on Gaussian splatting and a versatile training-free version utilizing neural fields and meshes. Our extensive experiments demonstrate that 3D-Adapter not only greatly enhances the geometry quality of text-to-multi-view models such as Instant3D and Zero123++, but also enables high-quality 3D generation using the plain text-to-image Stable Diffusion. Furthermore, we showcase the broad application potential of 3D-Adapter by presenting high quality results in text-to-3D, image-to-3D, text-to-texture, and text-to-avatar tasks.