Fuse3D: Generating 3D Assets Controlled by Multi-Image Fusion

TL;DR

Fuse3D is a pioneering method that enables the generation of 3D assets controlled by multiple images, allowing for detailed regional control and high-quality 3D structure synthesis through multi-condition fusion and local attention strategies.

Contribution

It introduces the first approach for controllable 3D asset generation using multiple condition images with innovative fusion and alignment techniques.

Findings

Successfully fuses multiple image regions into 3D assets

Achieves high-quality, region-specific 3D structures

Demonstrates flexible control over 3D asset details

Abstract

Recently, generating 3D assets with the control of condition images has achieved impressive quality. However, existing 3D generation methods are limited to handling a single control objective and lack the ability to utilize multiple images to independently control different regions of a 3D asset, which hinders their flexibility in applications. We propose Fuse3D, a novel method that enables generating 3D assets under the control of multiple images, allowing for the seamless fusion of multi-level regional controls from global views to intricate local details. First, we introduce a Multi-Condition Fusion Module to integrate the visual features from multiple image regions. Then, we propose a method to automatically align user-selected 2D image regions with their associated 3D regions based on semantic cues. Finally, to resolve control conflicts and enhance local control features from multi-condition images, we introduce a Local Attention Enhancement Strategy that flexibly balances region-specific feature fusion. Overall, we introduce the first method capable of controllable 3D asset generation from multiple condition images. The experimental results indicate that Fuse3D can flexibly fuse multiple 2D image regions into coherent 3D structures, resulting in high-quality 3D assets. Code and data for this paper are at https://jinnmnm.github.io/Fuse3d.github.io/.