STRICT: Stress Test of Rendering Images Containing Text

TL;DR

This paper introduces STRICT, a benchmark for evaluating diffusion models' ability to generate coherent, legible, and instruction-following text within images, revealing persistent limitations in current models.

Contribution

The paper presents a systematic benchmark to stress-test diffusion models' text rendering capabilities and analyzes their limitations in long-range consistency and instruction adherence.

Findings

Models struggle with long-range text coherence.

Persistent issues with text legibility and correctness.

Limitations in following complex text instructions.

Abstract

While diffusion models have revolutionized text-to-image generation with their ability to synthesize realistic and diverse scenes, they continue to struggle to generate consistent and legible text within images. This shortcoming is commonly attributed to the locality bias inherent in diffusion-based generation, which limits their ability to model long-range spatial dependencies. In this paper, we introduce $\textbf{STRICT}$, a benchmark designed to systematically stress-test the ability of diffusion models to render coherent and instruction-aligned text in images. Our benchmark evaluates models across multiple dimensions: (1) the maximum length of readable text that can be generated; (2) the correctness and legibility of the generated text, and (3) the ratio of not following instructions for generating text. We evaluate several state-of-the-art models, including proprietary and open-source variants, and reveal persistent limitations in long-range consistency and instruction-following capabilities. Our findings provide insights into architectural bottlenecks and motivate future research directions in multimodal generative modeling. We release our entire evaluation pipeline at https://github.com/tianyu-z/STRICT-Bench.