Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective

TL;DR

This paper introduces SemVarEffect and SemVarBench, new tools to evaluate how well text-to-image models handle semantic variations caused by word order changes, revealing the importance of cross-modal alignment.

Contribution

It proposes a novel causality-based evaluation metric and benchmark for assessing semantic variation handling in T2I models, highlighting the role of cross-modal alignment.

Findings

CogView-3-Plus and Ideogram 2 achieved top scores

Semantic variations in object relations are less understood than attributes

Cross-modal alignment significantly impacts handling semantic variations

Abstract

Accurate interpretation and visualization of human instructions are crucial for text-to-image (T2I) synthesis. However, current models struggle to capture semantic variations from word order changes, and existing evaluations, relying on indirect metrics like text-image similarity, fail to reliably assess these challenges. This often obscures poor performance on complex or uncommon linguistic patterns by the focus on frequent word combinations. To address these deficiencies, we propose a novel metric called SemVarEffect and a benchmark named SemVarBench, designed to evaluate the causality between semantic variations in inputs and outputs in T2I synthesis. Semantic variations are achieved through two types of linguistic permutations, while avoiding easily predictable literal variations. Experiments reveal that the CogView-3-Plus and Ideogram 2 performed the best, achieving a score of 0.2/1. Semantic variations in object relations are less understood than attributes, scoring 0.07/1 compared to 0.17-0.19/1. We found that cross-modal alignment in UNet or Transformers plays a crucial role in handling semantic variations, a factor previously overlooked by a focus on textual encoders. Our work establishes an effective evaluation framework that advances the T2I synthesis community's exploration of human instruction understanding. Our benchmark and code are available at https://github.com/zhuxiangru/SemVarBench .