Structured Captions Improve Prompt Adherence in Text-to-Image Models (Re-LAION-Caption 19M)

TL;DR

This paper demonstrates that enforcing a structured caption format during training improves prompt adherence and controllability in text-to-image models, reducing reliance on prompt engineering.

Contribution

We introduce Re-LAION-Caption 19M, a high-quality dataset with structured captions, and show that training with these captions enhances model alignment and controllability.

Findings

Structured captions lead to higher text-image alignment scores.

Training with structured captions improves model controllability.

The dataset is publicly available for further research.

Abstract

We argue that generative text-to-image models often struggle with prompt adherence due to the noisy and unstructured nature of large-scale datasets like LAION-5B. This forces users to rely heavily on prompt engineering to elicit desirable outputs. In this work, we propose that enforcing a consistent caption structure during training can significantly improve model controllability and alignment. We introduce Re-LAION-Caption 19M, a high-quality subset of Re-LAION-5B, comprising 19 million 1024x1024 images with captions generated by a Mistral 7B Instruct-based LLaVA-Next model. Each caption follows a four-part template: subject, setting, aesthetics, and camera details. We fine-tune PixArt-$\Sigma$ and Stable Diffusion 2 using both structured and randomly shuffled captions, and show that structured versions consistently yield higher text-image alignment scores using visual question answering (VQA) models. The dataset is publicly available at https://huggingface.co/datasets/supermodelresearch/Re-LAION-Caption19M.