Reading $\neq$ Seeing: Diagnosing and Closing the Typography Gap in Vision-Language Models

TL;DR

This paper investigates the typography recognition gap in vision-language models, revealing strengths in color detection but weaknesses in font style recognition, and proposes fine-tuning methods to improve performance.

Contribution

It systematically evaluates typography recognition in VLMs, identifies the limitations, and demonstrates effective fine-tuning strategies to enhance typographic understanding.

Findings

Color recognition is near-perfect across models.

Font style detection remains largely poor.

Fine-tuning improves font size recognition significantly.

Abstract

Vision-Language Models achieve near-perfect accuracy at reading text in images, yet prove largely typography-blind: capable of recognizing what text says, but not how it looks. We systematically investigate this gap by evaluating font family, size, style, and color recognition across 26 fonts, four scripts, and three difficulty levels. Our evaluation of 15 state-of-the-art VLMs reveals a striking perception hierarchy: color recognition is near-perfect, yet font style detection remains universally poor. We further find that model scale fails to predict performance and that accuracy is uniform across difficulty levels, together pointing to a training-data omission rather than a capacity ceiling. LoRA fine-tuning on a small set of synthetic samples substantially improves an open-source model, narrowing the gap to the best closed-source system and surpassing it on font size recognition. Font style alone remains resistant to fine-tuning, suggesting that relational visual reasoning may require architectural innovation beyond current patch-based encoders. We release our evaluation framework, data, and fine-tuning recipe to support progress in closing the typographic gap in vision-language understanding.