When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators

TL;DR

This study critically evaluates the effectiveness of modern text-to-image models as synthetic training data generators, revealing a decline in real-world classification performance despite visual improvements.

Contribution

It uncovers a hidden trend where newer T2I models produce less diverse data, challenging assumptions about their usefulness for training classifiers.

Findings

Synthetic datasets from newer T2I models lead to lower accuracy on real data.

Modern T2I models tend to generate narrow, aesthetic-centric images.

Progress in visual fidelity does not equate to improved data realism.

Abstract

Recent text-to-image (T2I) diffusion models produce visually stunning images and demonstrate excellent prompt following. But do they perform well as synthetic vision data generators? In this work, we revisit the promise of synthetic data as a scalable substitute for real training sets and uncover a surprising performance regression. We generate large-scale synthetic datasets using state-of-the-art T2I models released between 2022 and 2025, train standard classifiers solely on this synthetic data, and evaluate them on real test data. Despite observable advances in visual fidelity and prompt adherence, classification accuracy on real test data consistently declines with newer T2I models as training data generators. Our analysis reveals a hidden trend: These models collapse to a narrow, aesthetic-centric distribution that undermines diversity and real data distribution coverage. Overall, our findings challenge a growing assumption in vision research, namely that progress in generative realism implies progress in data realism. We thus highlight an urgent need to rethink the capabilities of modern T2I models as reliable training data generators.