Zero-Shot Synthetic-to-Real Handwritten Text Recognition via Task Analogies

TL;DR

This paper introduces a zero-shot method for improving handwritten text recognition on real data by transferring learned parameter adjustments from synthetic to real handwriting across multiple languages.

Contribution

It proposes a novel transfer learning approach that leverages source languages and linguistic similarity to enhance recognition in unseen target languages without real target data.

Findings

Consistent improvements over synthetic-only baselines across five languages.

Transferring corrections benefits even unrelated languages.

Method works across six different architectures.

Abstract

Handwritten Text Recognition (HTR) models trained on synthetic handwriting often struggle to generalize to real text, and existing adaptation methods still require real samples from the target domain. In this work, we tackle the fully zero-shot synthetic-to-real generalization setting, where no real data from the target language is available. Our approach learns how model parameters change when moving from synthetic to real handwriting in one or more source languages and transfers this learned correction to new target languages. When using multiple sources, we rely on linguistic similarity to weigh their contrubition when combining them. Experiments across five languages and six architectures show consistent improvements over synthetic-only baselines and reveal that the transferred corrections benefit even languages unrelated to the sources.