DiffBlender: Composable and Versatile Multimodal Text-to-Image Diffusion Models

TL;DR

DiffBlender introduces a unified multimodal diffusion model that effectively combines structure, layout, and attribute inputs for versatile and high-quality text-to-image generation without extensive retraining.

Contribution

It presents a novel framework capable of integrating multiple modalities into a single diffusion model with minimal parameter updates, setting new benchmarks in multimodal image synthesis.

Findings

Achieves state-of-the-art results in multimodal generation benchmarks.

Successfully integrates multiple modalities without modifying pre-trained diffusion models.

Demonstrates diverse applications in detailed image synthesis.

Abstract

In this study, we aim to enhance the capabilities of diffusion-based text-to-image (T2I) generation models by integrating diverse modalities beyond textual descriptions within a unified framework. To this end, we categorize widely used conditional inputs into three modality types: structure, layout, and attribute. We propose a multimodal T2I diffusion model, which is capable of processing all three modalities within a single architecture without modifying the parameters of the pre-trained diffusion model, as only a small subset of components is updated. Our approach sets new benchmarks in multimodal generation through extensive quantitative and qualitative comparisons with existing conditional generation methods. We demonstrate that DiffBlender effectively integrates multiple sources of information and supports diverse applications in detailed image synthesis. The code and demo are available at https://github.com/sungnyun/diffblender.