WordCon: Word-level Typography Control in Scene Text Rendering

TL;DR

WordCon introduces a novel framework for precise word-level typography control in scene text rendering, combining a new dataset, cross-modal alignment, and an efficient fine-tuning method to improve controllability and versatility.

Contribution

The paper presents a new dataset, the TIA framework, and WordCon, a hybrid PEFT method, enabling accurate and efficient word-level control in scene text image generation.

Findings

Outperforms state-of-the-art methods in controllability and quality.

Demonstrates effectiveness across artistic, editing, and image-conditioned text rendering.

Provides publicly available datasets and source code for further research.

Abstract

Achieving precise word-level typography control within generated images remains a persistent challenge. To address it, we newly construct a word-level controlled scene text dataset and introduce the Text-Image Alignment (TIA) framework. This framework leverages cross-modal correspondence between text and local image regions provided by grounding models to enhance the Text-to-Image (T2I) model training. Furthermore, we propose WordCon, a hybrid parameter-efficient fine-tuning (PEFT) method. WordCon reparameterizes selective key parameters, improving both efficiency and portability. This allows seamless integration into diverse pipelines, including artistic text rendering, text editing, and image-conditioned text rendering. To further enhance controllability, the masked loss at the latent level is applied to guide the model to concentrate on learning the text region in the image, and the joint-attention loss provides feature-level supervision to promote disentanglement between different words. Both qualitative and quantitative results demonstrate the superiority of our method to the state of the art. The datasets and source code will be available for academic use.