Prompt fidelity of ChatGPT4o / Dall-E3 text-to-image visualisations

TL;DR

This paper evaluates how accurately ChatGPT4o and DALL-E3 translate detailed prompts into images, revealing a 15.6% deviation rate with specific biases in rendering personal attributes and paraphernalia.

Contribution

It provides a systematic analysis of prompt fidelity in state-of-the-art text-to-image models using real-world datasets, highlighting specific areas of deviation and bias.

Findings

DALL-E3 deviates from prompts in 15.6% of attributes

Errors are lowest for paraphernalia, higher for appearance, highest for personal attributes

Prompt-to-image fidelity gaps have implications for bias detection and model evaluation

Abstract

This study examines the prompt fidelity of ChatGPT4o / DALL-E3 text-to-image visualisations by analysing whether attributes explicitly specified in autogenously generated prompts are correctly rendered in the resulting images. Using two public-domain datasets comprising 200 visualisations of women working in the cultural and creative industries and 230 visualisations of museum curators, the study assessed accuracy across personal attributes (age, hair), appearance (attire, glasses), and paraphernalia (name tags, clipboards). While correctly rendered in most cases, DALL-E3 deviated from prompt specifications in 15.6% of all attributes (n=710). Errors were lowest for paraphernalia, moderate for personal appearance, and highest for depictions of the person themselves, particularly age. These findings demonstrate measurable prompt-to-image fidelity gaps with implications for bias detection and model evaluation.