One-Shot Diffusion Mimicker for Handwritten Text Generation

TL;DR

This paper introduces One-DM, a diffusion-based model that can generate handwritten text mimicking a specific style from just a single sample, outperforming previous multi-sample methods.

Contribution

The paper presents a novel one-shot style extraction technique using high-frequency features and integrates it into a diffusion model for high-quality handwritten text generation.

Findings

Effective style transfer from one sample in multiple languages

Outperforms methods requiring over ten samples

High-quality, realistic handwritten text generation

Abstract

Existing handwritten text generation methods often require more than ten handwriting samples as style references. However, in practical applications, users tend to prefer a handwriting generation model that operates with just a single reference sample for its convenience and efficiency. This approach, known as "one-shot generation", significantly simplifies the process but poses a significant challenge due to the difficulty of accurately capturing a writer's style from a single sample, especially when extracting fine details from the characters' edges amidst sparse foreground and undesired background noise. To address this problem, we propose a One-shot Diffusion Mimicker (One-DM) to generate handwritten text that can mimic any calligraphic style with only one reference sample. Inspired by the fact that high-frequency information of the individual sample often contains distinct style patterns (e.g., character slant and letter joining), we develop a novel style-enhanced module to improve the style extraction by incorporating high-frequency components from a single sample. We then fuse the style features with the text content as a merged condition for guiding the diffusion model to produce high-quality handwritten text images. Extensive experiments demonstrate that our method can successfully generate handwriting scripts with just one sample reference in multiple languages, even outperforming previous methods using over ten samples. Our source code is available at https://github.com/dailenson/One-DM.