Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model

TL;DR

This paper introduces Diff-Text, a training-free diffusion-based framework capable of generating realistic multilingual scene text images from any language, improving text accuracy and image naturalness.

Contribution

The paper presents a novel multilingual scene text generation method using diffusion models with localized attention constraints and contrastive prompts, without requiring additional training.

Findings

Outperforms existing methods in text recognition accuracy

Enhances naturalness of foreground-background blending

Effective in generating multilingual scene text images

Abstract

Recently, diffusion-based image generation methods are credited for their remarkable text-to-image generation capabilities, while still facing challenges in accurately generating multilingual scene text images. To tackle this problem, we propose Diff-Text, which is a training-free scene text generation framework for any language. Our model outputs a photo-realistic image given a text of any language along with a textual description of a scene. The model leverages rendered sketch images as priors, thus arousing the potential multilingual-generation ability of the pre-trained Stable Diffusion. Based on the observation from the influence of the cross-attention map on object placement in generated images, we propose a localized attention constraint into the cross-attention layer to address the unreasonable positioning problem of scene text. Additionally, we introduce contrastive image-level prompts to further refine the position of the textual region and achieve more accurate scene text generation. Experiments demonstrate that our method outperforms the existing method in both the accuracy of text recognition and the naturalness of foreground-background blending.