On the Generalization of Handwritten Text Recognition Models

TL;DR

This paper investigates how handwritten text recognition models perform on out-of-distribution data, revealing key factors affecting generalization and providing insights for future improvements in real-world applications.

Contribution

It analyzes the limitations of HTR models in domain generalization without prior access to OOD data, highlighting the impact of textual and visual divergence.

Findings

Textual divergence is the most significant factor for generalization.

Model errors in OOD scenarios can be estimated with less than 10 points discrepancy in 70% cases.

Synthetic data usage influences HTR model generalization.

Abstract

Recent advances in Handwritten Text Recognition (HTR) have led to significant reductions in transcription errors on standard benchmarks under the i.i.d. assumption, thus focusing on minimizing in-distribution (ID) errors. However, this assumption does not hold in real-world applications, which has motivated HTR research to explore Transfer Learning and Domain Adaptation techniques. In this work, we investigate the unaddressed limitations of HTR models in generalizing to out-of-distribution (OOD) data. We adopt the challenging setting of Domain Generalization, where models are expected to generalize to OOD data without any prior access. To this end, we analyze 336 OOD cases from eight state-of-the-art HTR models across seven widely used datasets, spanning five languages. Additionally, we study how HTR models leverage synthetic data to generalize. We reveal that the most significant factor for generalization lies in the textual divergence between domains, followed by visual divergence. We demonstrate that the error of HTR models in OOD scenarios can be reliably estimated, with discrepancies falling below 10 points in 70\% of cases. We identify the underlying limitations of HTR models, laying the foundation for future research to address this challenge.