Self-Prompting Diffusion Transformer for Open-Vocabulary Scene Text Editing via In-Context Learning

TL;DR

This paper introduces a self-prompting diffusion transformer for open-vocabulary scene text editing that constructs style and glyph prompts directly from images, achieving state-of-the-art results without extra encoders.

Contribution

It proposes a novel self-prompting method using in-image prompts and a two-stage training strategy for improved scene text editing.

Findings

Achieves state-of-the-art performance in text accuracy and style consistency.

Supports open-vocabulary text editing across various languages.

Eliminates the need for additional style or glyph encoders.

Abstract

Scene text editing aims to modify text in a target region of an image while preserving surrounding background style and texture. Existing methods rely solely on image background information while neglecting the visual details of target regions, which discards stylistic features in the original text and essentially degrades the task to text rendering. Moreover, the conditions imposed by pre-trained glyph encoder limit the scope of editable text. To address these issues, this paper proposes a self-prompting scene text editing method that constructs style and glyph prompts directly from the original image, without introducing additional style or glyph encoders. We employ a two-stage training strategy: the diffusion transformer is first trained on large-scale self-supervised data and then refined using a small set of paired images. By leveraging the in-context learning capability of the Multi-Modal Diffusion Transformer (MM-DiT), it achieves open-vocabulary and style-consistent text editing. Experimental results on various languages demonstrate that our method achieves the state-of-the-art performance in both text accuracy and style consistency. Our project page: \href{https://hongxiii.github.io/mstedit}{hongxiii.github.io/mstedit}.