CEIDM: A Controlled Entity and Interaction Diffusion Model for Enhanced Text-to-Image Generation

TL;DR

CEIDM introduces a diffusion-based text-to-image generation method with dual controls for entities and their interactions, leveraging LLMs and advanced clustering to produce more realistic and semantically accurate images.

Contribution

The paper presents a novel diffusion model with dual control mechanisms for entities and interactions, utilizing LLM-based relationship mining and interactive action clustering.

Findings

Outperforms existing methods in entity control accuracy.

Produces images with more realistic interactive relationships.

Enhances semantic understanding of actions in generated images.

Abstract

In Text-to-Image (T2I) generation, the complexity of entities and their intricate interactions pose a significant challenge for T2I method based on diffusion model: how to effectively control entity and their interactions to produce high-quality images. To address this, we propose CEIDM, a image generation method based on diffusion model with dual controls for entity and interaction. First, we propose an entity interactive relationships mining approach based on Large Language Models (LLMs), extracting reasonable and rich implicit interactive relationships through chain of thought to guide diffusion models to generate high-quality images that are closer to realistic logic and have more reasonable interactive relationships. Furthermore, We propose an interactive action clustering and offset method to cluster and offset the interactive action features contained in each text prompts. By constructing global and local bidirectional offsets, we enhance semantic understanding and detail supplementation of original actions, making the model's understanding of the concept of interactive "actions" more accurate and generating images with more accurate interactive actions. Finally, we design an entity control network which generates masks with entity semantic guidance, then leveraging multi-scale convolutional network to enhance entity feature and dynamic network to fuse feature. It effectively controls entities and significantly improves image quality. Experiments show that the proposed CEIDM method is better than the most representative existing methods in both entity control and their interaction control.