DreamComposer++: Empowering Diffusion Models with Multi-View Conditions for 3D Content Generation

TL;DR

DreamComposer++ enhances diffusion-based 3D content generation by integrating multi-view conditions, enabling controllable and high-quality novel view synthesis through a view-aware 3D lifting and feature fusion framework.

Contribution

It introduces a scalable framework that incorporates multi-view conditions into diffusion models, improving controllability and quality in 3D content generation.

Findings

Improves controllability of novel view synthesis.

Seamlessly integrates with existing diffusion models.

Enhances 3D object reconstruction capabilities.

Abstract

Recent advancements in leveraging pre-trained 2D diffusion models achieve the generation of high-quality novel views from a single in-the-wild image. However, existing works face challenges in producing controllable novel views due to the lack of information from multiple views. In this paper, we present DreamComposer++, a flexible and scalable framework designed to improve current view-aware diffusion models by incorporating multi-view conditions. Specifically, DreamComposer++ utilizes a view-aware 3D lifting module to extract 3D representations of an object from various views. These representations are then aggregated and rendered into the latent features of target view through the multi-view feature fusion module. Finally, the obtained features of target view are integrated into pre-trained image or video diffusion models for novel view synthesis. Experimental results demonstrate that DreamComposer++ seamlessly integrates with cutting-edge view-aware diffusion models and enhances their abilities to generate controllable novel views from multi-view conditions. This advancement facilitates controllable 3D object reconstruction and enables a wide range of applications.