SPDiffusion: Semantic Protection Diffusion Models for Multi-concept Text-to-image Generation

TL;DR

SPDiffusion is a novel diffusion model that reduces semantic entanglement in multi-concept text-to-image generation by protecting concept regions, leading to more accurate and consistent images.

Contribution

The paper introduces SPDiffusion, a new framework with region extraction and attention protection techniques to improve multi-concept image generation.

Findings

Achieves state-of-the-art results on benchmarks.

Effectively reduces concept entanglement and attribute misbinding.

Enhances text-image consistency in multi-concept scenarios.

Abstract

Recent text-to-image models have achieved impressive results in generating high-quality images. However, when tasked with multi-concept generation creating images that contain multiple characters or objects, existing methods often suffer from semantic entanglement, including concept entanglement and improper attribute binding, leading to significant text-image inconsistency. We identify that semantic entanglement arises when certain regions of the latent features attend to incorrect concept and attribute tokens. In this work, we propose the Semantic Protection Diffusion Model (SPDiffusion) to address both concept entanglement and improper attribute binding using only a text prompt as input. The SPDiffusion framework introduces a novel concept region extraction method SP-Extraction to resolve region entanglement in cross-attention, along with SP-Attn, which protects concept regions from the influence of irrelevant attributes and concepts. To evaluate our method, we test it on existing benchmarks, where SPDiffusion achieves state-of-the-art results, demonstrating its effectiveness.