TypeScore: A Text Fidelity Metric for Text-to-Image Generative Models

TL;DR

TypeScore is a new evaluation metric that accurately measures a text-to-image model's ability to generate images with high-fidelity embedded text, providing finer discrimination than existing metrics like CLIPScore.

Contribution

This work introduces TypeScore, a novel metric that assesses the fidelity of embedded text in generated images, enhancing evaluation sensitivity for instruction-following capabilities.

Findings

TypeScore outperforms CLIPScore in differentiating models based on embedded text fidelity.

The metric effectively evaluates stylistic adherence in image generation.

Human studies validate the effectiveness of TypeScore as an evaluation tool.

Abstract

Evaluating text-to-image generative models remains a challenge, despite the remarkable progress being made in their overall performances. While existing metrics like CLIPScore work for coarse evaluations, they lack the sensitivity to distinguish finer differences as model performance rapidly improves. In this work, we focus on the text rendering aspect of these models, which provides a lens for evaluating a generative model's fine-grained instruction-following capabilities. To this end, we introduce a new evaluation framework called TypeScore to sensitively assess a model's ability to generate images with high-fidelity embedded text by following precise instructions. We argue that this text generation capability serves as a proxy for general instruction-following ability in image synthesis. TypeScore uses an additional image description model and leverages an ensemble dissimilarity measure between the original and extracted text to evaluate the fidelity of the rendered text. Our proposed metric demonstrates greater resolution than CLIPScore to differentiate popular image generation models across a range of instructions with diverse text styles. Our study also evaluates how well these vision-language models (VLMs) adhere to stylistic instructions, disentangling style evaluation from embedded-text fidelity. Through human evaluation studies, we quantitatively meta-evaluate the effectiveness of the metric. Comprehensive analysis is conducted to explore factors such as text length, captioning models, and current progress towards human parity on this task. The framework provides insights into remaining gaps in instruction-following for image generation with embedded text.