Multimodal Benchmarking and Recommendation of Text-to-Image Generation Models

TL;DR

This paper introduces an open-source benchmarking framework for text-to-image models, emphasizing metadata-augmented prompts, and demonstrates how structured metadata improves image quality and model robustness.

Contribution

It provides a unified evaluation framework using diverse metrics and shows the benefits of metadata enrichment for text-to-image generation.

Findings

Metadata augmentation improves visual realism.

Structured prompts enhance semantic fidelity.

Framework enables task-specific model recommendations.

Abstract

This work presents an open-source unified benchmarking and evaluation framework for text-to-image generation models, with a particular focus on the impact of metadata augmented prompts. Leveraging the DeepFashion-MultiModal dataset, we assess generated outputs through a comprehensive set of quantitative metrics, including Weighted Score, CLIP (Contrastive Language Image Pre-training)-based similarity, LPIPS (Learned Perceptual Image Patch Similarity), FID (Frechet Inception Distance), and retrieval-based measures, as well as qualitative analysis. Our results demonstrate that structured metadata enrichments greatly enhance visual realism, semantic fidelity, and model robustness across diverse text-to-image architectures. While not a traditional recommender system, our framework enables task-specific recommendations for model selection and prompt design based on evaluation metrics.