Improving Multi-Subject Consistency in Open-Domain Image Generation with Isolation and Reposition Attention

TL;DR

This paper introduces IR-Diffusion, a training-free diffusion model with Isolation and Reposition Attention, significantly improving multi-subject consistency in open-domain image generation by addressing internal attraction and positional referencing issues.

Contribution

The paper proposes a novel IR-Diffusion model with Isolation and Reposition Attention to enhance multi-subject consistency without additional training.

Findings

IR-Diffusion outperforms existing methods in multi-subject consistency.

Isolation Attention prevents subjects from converging into a single entity.

Reposition Attention aligns subjects to improve referencing accuracy.

Abstract

Training-free diffusion models have achieved remarkable progress in generating multi-subject consistent images within open-domain scenarios. The key idea of these methods is to incorporate reference subject information within the attention layer. However, existing methods still obtain suboptimal performance when handling numerous subjects. This paper reveals two primary issues contributing to this deficiency. Firstly, the undesired internal attraction between different subjects within the target image can lead to the convergence of multiple subjects into a single entity. Secondly, tokens tend to reference nearby tokens, which reduces the effectiveness of the attention mechanism when there is a significant positional difference between subjects in reference and target images. To address these issues, we propose a training-free diffusion model with Isolation and Reposition Attention, named IR-Diffusion. Specifically, Isolation Attention ensures that multiple subjects in the target image do not reference each other, effectively eliminating the subject convergence. On the other hand, Reposition Attention involves scaling and repositioning subjects in both reference and target images to the same position within the images. This ensures that subjects in the target image can better reference those in the reference image, thereby maintaining better consistency. Extensive experiments demonstrate that IR-Diffusion significantly enhances multi-subject consistency, outperforming all existing methods in open-domain scenarios.