VLEU: a Method for Automatic Evaluation for Generalizability of Text-to-Image Models

TL;DR

VLEU is a novel evaluation metric for Text-to-Image models that measures their ability to handle diverse prompts by analyzing the distribution of generated images relative to input texts using large language models and CLIP.

Contribution

We introduce VLEU, a new metric leveraging large language models and CLIP to assess the generalizability of T2I models across diverse prompts, filling a gap in existing evaluation methods.

Findings

VLEU effectively measures T2I model generalization.

VLEU correlates well with model finetuning improvements.

VLEU distinguishes different T2I models based on prompt diversity.

Abstract

Progress in Text-to-Image (T2I) models has significantly improved the generation of images from textual descriptions. However, existing evaluation metrics do not adequately assess the models' ability to handle a diverse range of textual prompts, which is crucial for their generalizability. To address this, we introduce a new metric called Visual Language Evaluation Understudy (VLEU). VLEU uses large language models to sample from the visual text domain, the set of all possible input texts for T2I models, to generate a wide variety of prompts. The images generated from these prompts are evaluated based on their alignment with the input text using the CLIP model.VLEU quantifies a model's generalizability by computing the Kullback-Leibler divergence between the marginal distribution of the visual text and the conditional distribution of the images generated by the model. This metric provides a quantitative way to compare different T2I models and track improvements during model finetuning. Our experiments demonstrate the effectiveness of VLEU in evaluating the generalization capability of various T2I models, positioning it as an essential metric for future research in text-to-image synthesis.