TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generation

TL;DR

TweedieMix is a novel inference-time method that enhances multi-concept fusion in diffusion models, improving the fidelity of personalized image and video generation involving multiple concepts.

Contribution

It introduces a two-stage sampling process that ensures inclusion and blending of multiple personalized concepts during diffusion-based generation.

Findings

Higher fidelity in multi-concept generation compared to existing methods

Effective extension to image-to-video diffusion models

Demonstrated success in generating personalized videos with multiple concepts

Abstract

Despite significant advancements in customizing text-to-image and video generation models, generating images and videos that effectively integrate multiple personalized concepts remains a challenging task. To address this, we present TweedieMix, a novel method for composing customized diffusion models during the inference phase. By analyzing the properties of reverse diffusion sampling, our approach divides the sampling process into two stages. During the initial steps, we apply a multiple object-aware sampling technique to ensure the inclusion of the desired target objects. In the later steps, we blend the appearances of the custom concepts in the de-noised image space using Tweedie's formula. Our results demonstrate that TweedieMix can generate multiple personalized concepts with higher fidelity than existing methods. Moreover, our framework can be effortlessly extended to image-to-video diffusion models, enabling the generation of videos that feature multiple personalized concepts. Results and source code are in our anonymous project page.