How to Train your Text-to-Image Model: Evaluating Design Choices for Synthetic Training Captions

TL;DR

This paper systematically investigates how different synthetic captioning strategies affect the performance of text-to-image models, revealing trade-offs between caption quality, diversity, and output bias.

Contribution

It provides the first comprehensive analysis of synthetic caption design choices and their impact on model performance and output characteristics.

Findings

High-quality dense captions improve text alignment.

Randomized caption lengths balance aesthetics and diversity.

Caption distribution shifts significantly affect output bias.

Abstract

Training data is at the core of any successful text-to-image models. The quality and descriptiveness of image text are crucial to a model's performance. Given the noisiness and inconsistency in web-scraped datasets, recent works shifted towards synthetic training captions. While this setup is generally believed to produce more capable models, current literature does not provide any insights into its design choices. This study closes this gap by systematically investigating how different synthetic captioning strategies impact the downstream performance of text-to-image models. Our experiments demonstrate that dense, high-quality captions enhance text alignment but may introduce trade-offs in output aesthetics and diversity. Conversely, captions of randomized lengths yield balanced improvements across aesthetics and alignment without compromising sample diversity. We also demonstrate that varying caption distributions introduce significant shifts in the output bias of a trained model. Our findings underscore the importance of caption design in achieving optimal model performance and provide practical insights for more effective training data strategies in text-to-image generation.