HTR-ConvText: Leveraging Convolution and Textual Information for Handwritten Text Recognition

TL;DR

HTR-ConvText is a novel model that combines convolutional and textual features to improve handwritten text recognition, especially with limited data and diverse handwriting styles.

Contribution

The paper introduces HTR-ConvText, a hybrid architecture integrating CNN, MobileViT, and hierarchical encoding to enhance feature extraction and generalization in handwritten text recognition.

Findings

Achieves better accuracy on multiple datasets.

Improves generalization with limited training data.

Outperforms existing methods in complex handwriting scenarios.

Abstract

Handwritten Text Recognition remains challenging due to the limited data, high writing style variance, and scripts with complex diacritics. Existing approaches, though partially address these issues, often struggle to generalize without massive synthetic data. To address these challenges, we propose HTR-ConvText, a model designed to capture fine-grained, stroke-level local features while preserving global contextual dependencies. In the feature extraction stage, we integrate a residual Convolutional Neural Network backbone with a MobileViT with Positional Encoding block. This enables the model to both capture structural patterns and learn subtle writing details. We then introduce the ConvText encoder, a hybrid architecture combining global context and local features within a hierarchical structure that reduces sequence length for improved efficiency. Additionally, an auxiliary module injects textual context to mitigate the weakness of Connectionist Temporal Classification. Evaluations on IAM, READ2016, LAM and HANDS-VNOnDB demonstrate that our approach achieves improved performance and better generalization compared to existing methods, especially in scenarios with limited training samples and high handwriting diversity.