Selectively Informative Description can Reduce Undesired Embedding Entanglements in Text-to-Image Personalization

TL;DR

This paper introduces SID, a novel text description strategy using multimodal GPT-4, to reduce undesired embedding entanglements in text-to-image personalization, improving image quality and alignment.

Contribution

The paper proposes SID, a new method for generating informative descriptions that mitigate biases and entanglements in text-to-image models, enhancing personalization fidelity.

Findings

SID reduces bias reflection in generated images.

Improves alignment between images and prompts.

Analyzes attention maps to understand bias mitigation.

Abstract

In text-to-image personalization, a timely and crucial challenge is the tendency of generated images overfitting to the biases present in the reference images. We initiate our study with a comprehensive categorization of the biases into background, nearby-object, tied-object, substance (in style re-contextualization), and pose biases. These biases manifest in the generated images due to their entanglement into the subject embedding. This undesired embedding entanglement not only results in the reflection of biases from the reference images into the generated images but also notably diminishes the alignment of the generated images with the given generation prompt. To address this challenge, we propose SID~(Selectively Informative Description), a text description strategy that deviates from the prevalent approach of only characterizing the subject's class identification. SID is generated utilizing multimodal GPT-4 and can be seamlessly integrated into optimization-based models. We present comprehensive experimental results along with analyses of cross-attention maps, subject-alignment, non-subject-disentanglement, and text-alignment.