Semi-Supervised Adaptation of Diffusion Models for Handwritten Text Generation

TL;DR

This paper introduces a semi-supervised diffusion model for handwritten text generation that adapts to new styles using unlabeled data, improving the synthesis of realistic, styled handwritten words for training data augmentation.

Contribution

It extends latent diffusion models with style conditioning via a masked autoencoder and proposes a semi-supervised training scheme for style adaptation in handwritten text generation.

Findings

Improved handwriting style transfer and generation quality.

Effective adaptation to new, unseen handwriting styles.

Enhanced training data for downstream handwriting recognition models.

Abstract

The generation of images of realistic looking, readable handwritten text is a challenging task which is referred to as handwritten text generation (HTG). Given a string and examples from a writer, the goal is to synthesize an image depicting the correctly spelled word in handwriting with the calligraphic style of the desired writer. An important application of HTG is the generation of training images in order to adapt downstream models for new data sets. With their success in natural image generation, diffusion models (DMs) have become the state-of-the-art approach in HTG. In this work, we present an extension of a latent DM for HTG to enable generation of writing styles not seen during training by learning style conditioning with a masked auto encoder. Our proposed content encoder allows for different ways of conditioning the DM on textual and calligraphic features. Additionally, we employ classifier-free guidance and explore the influence on the quality of the generated training images. For adapting the model to a new unlabeled data set, we propose a semi-supervised training scheme. We evaluate our approach on the IAM-database and use the RIMES-database to examine the generation of data not seen during training achieving improvements in this particularly promising application of DMs for HTG.