DreamRelation: Bridging Customization and Relation Generation

TL;DR

DreamRelation is a novel framework that enhances customized image generation by accurately modeling object identities and relationships, addressing pose adjustments and object confusion issues.

Contribution

It introduces a relation-aware generation framework with keypoint matching and local feature modules, improving relation accuracy and object distinction in generated images.

Findings

Outperforms existing models in relation accuracy and identity preservation

Effectively handles pose adjustments and object overlaps

Demonstrates superior results on new relation-specific benchmarks

Abstract

Customized image generation is essential for creating personalized content based on user prompts, allowing large-scale text-to-image diffusion models to more effectively meet individual needs. However, existing models often neglect the relationships between customized objects in generated images. In contrast, this work addresses this gap by focusing on relation-aware customized image generation, which seeks to preserve the identities from image prompts while maintaining the relationship specified in text prompts. Specifically, we introduce DreamRelation, a framework that disentangles identity and relation learning using a carefully curated dataset. Our training data consists of relation-specific images, independent object images containing identity information, and text prompts to guide relation generation. Then, we propose two key modules to tackle the two main challenges: generating accurate and natural relationships, especially when significant pose adjustments are required, and avoiding object confusion in cases of overlap. First, we introduce a keypoint matching loss that effectively guides the model in adjusting object poses closely tied to their relationships. Second, we incorporate local features of the image prompts to better distinguish between objects, preventing confusion in overlapping cases. Extensive results on our proposed benchmarks demonstrate the superiority of DreamRelation in generating precise relations while preserving object identities across a diverse set of objects and relationships.