DiffUTE: Universal Text Editing Diffusion Model

TL;DR

DiffUTE is a universal, self-supervised diffusion model for realistic, multilingual text editing in images, improving rendering accuracy and style control by leveraging web data and specialized network modifications.

Contribution

The paper introduces a novel self-supervised diffusion model with network modifications for multilingual text editing in images, enhancing realism and controllability.

Findings

Achieves high-fidelity, controllable text editing in diverse images.

Effectively handles multilingual characters with glyph and position info.

Leverages large-scale web data for improved model representation.

Abstract

Diffusion model based language-guided image editing has achieved great success recently. However, existing state-of-the-art diffusion models struggle with rendering correct text and text style during generation. To tackle this problem, we propose a universal self-supervised text editing diffusion model (DiffUTE), which aims to replace or modify words in the source image with another one while maintaining its realistic appearance. Specifically, we build our model on a diffusion model and carefully modify the network structure to enable the model for drawing multilingual characters with the help of glyph and position information. Moreover, we design a self-supervised learning framework to leverage large amounts of web data to improve the representation ability of the model. Experimental results show that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity. Our code will be avaliable in \url{https://github.com/chenhaoxing/DiffUTE}.