Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generation

TL;DR

This paper improves text-to-image models based on MMDiT by addressing subject neglect and mixing issues through test-time optimization and novel loss functions, resulting in higher quality and success rates in generating similar subjects.

Contribution

We identify key ambiguities in MMDiT causing subject mixing and propose on-the-fly latent repair with tailored loss functions and strategies to enhance generation quality.

Findings

Significant improvement in generation quality on a challenging dataset

Higher success rates compared to existing methods

Effective mitigation of subject ambiguity issues

Abstract

Representing the cutting-edge technique of text-to-image models, the latest Multimodal Diffusion Transformer (MMDiT) largely mitigates many generation issues existing in previous models. However, we discover that it still suffers from subject neglect or mixing when the input text prompt contains multiple subjects of similar semantics or appearance. We identify three possible ambiguities within the MMDiT architecture that cause this problem: Inter-block Ambiguity, Text Encoder Ambiguity, and Semantic Ambiguity. To address these issues, we propose to repair the ambiguous latent on-the-fly by test-time optimization at early denoising steps. In detail, we design three loss functions: Block Alignment Loss, Text Encoder Alignment Loss, and Overlap Loss, each tailored to mitigate these ambiguities. Despite significant improvements, we observe that semantic ambiguity persists when generating multiple similar subjects, as the guidance provided by overlap loss is not explicit enough. Therefore, we further propose Overlap Online Detection and Back-to-Start Sampling Strategy to alleviate the problem. Experimental results on a newly constructed challenging dataset of similar subjects validate the effectiveness of our approach, showing superior generation quality and much higher success rates over existing methods. Our code will be available at https://github.com/wtybest/EnMMDiT.