MetaWriter: Personalized Handwritten Text Recognition Using Meta-Learned Prompt Tuning

TL;DR

MetaWriter introduces a prompt tuning approach with meta-learning for personalized handwritten text recognition, enabling style adaptation with minimal parameter updates and unlabeled data, outperforming prior methods on standard benchmarks.

Contribution

The paper presents a novel prompt tuning framework with meta-learning for efficient personalization in handwritten text recognition, reducing parameter updates and annotation needs.

Findings

Outperforms previous state-of-the-art on RIMES and IAM datasets.

Uses less than 1% of model parameters for personalization.

Achieves 20x fewer parameters compared to existing methods.

Abstract

Recent advancements in handwritten text recognition (HTR) have enabled the effective conversion of handwritten text to digital formats. However, achieving robust recognition across diverse writing styles remains challenging. Traditional HTR methods lack writer-specific personalization at test time due to limitations in model architecture and training strategies. Existing attempts to bridge this gap, through gradient-based meta-learning, still require labeled examples and suffer from parameter-inefficient fine-tuning, leading to substantial computational and memory overhead. To overcome these challenges, we propose an efficient framework that formulates personalization as prompt tuning, incorporating an auxiliary image reconstruction task with a self-supervised loss to guide prompt adaptation with unlabeled test-time examples. To ensure self-supervised loss effectively minimizes text recognition error, we leverage meta-learning to learn the optimal initialization of the prompts. As a result, our method allows the model to efficiently capture unique writing styles by updating less than 1% of its parameters and eliminating the need for time-intensive annotation processes. We validate our approach on the RIMES and IAM Handwriting Database benchmarks, where it consistently outperforms previous state-of-the-art methods while using 20x fewer parameters. We believe this represents a significant advancement in personalized handwritten text recognition, paving the way for more reliable and practical deployment in resource-constrained scenarios.