A Cat Is A Cat (Not A Dog!): Unraveling Information Mix-ups in Text-to-Image Encoders through Causal Analysis and Embedding Optimization

TL;DR

This paper investigates how text embeddings influence image generation in T2I models, revealing biases and information loss, and proposes a training-free optimization method and a new evaluation metric to improve and measure information balance.

Contribution

It introduces a novel, training-free text embedding balance optimization method and a new automatic evaluation metric for better assessment of information retention in T2I models.

Findings

125.42% improvement in information balance with the proposed method

81% concordance of the new metric with human judgments

Enhanced object presence and accuracy in generated images

Abstract

This paper analyzes the impact of causal manner in the text encoder of text-to-image (T2I) diffusion models, which can lead to information bias and loss. Previous works have focused on addressing the issues through the denoising process. However, there is no research discussing how text embedding contributes to T2I models, especially when generating more than one object. In this paper, we share a comprehensive analysis of text embedding: i) how text embedding contributes to the generated images and ii) why information gets lost and biases towards the first-mentioned object. Accordingly, we propose a simple but effective text embedding balance optimization method, which is training-free, with an improvement of 125.42% on information balance in stable diffusion. Furthermore, we propose a new automatic evaluation metric that quantifies information loss more accurately than existing methods, achieving 81% concordance with human assessments. This metric effectively measures the presence and accuracy of objects, addressing the limitations of current distribution scores like CLIP's text-image similarities.