DiffSTR: Controlled Diffusion Models for Scene Text Removal

TL;DR

DiffSTR introduces a diffusion model-based approach for scene text removal, leveraging advanced inpainting and mask refinement techniques to produce high-quality, natural, text-free images while addressing common challenges like artifacts and inconsistent textures.

Contribution

This work pioneers the use of diffusion models with a novel mask refinement pipeline for scene text removal, improving accuracy and image quality over prior methods.

Findings

Outperforms state-of-the-art methods on SCUT-EnsText and SCUT-Syn datasets

Achieves more accurate inpainting masks with the segmentation-based refinement

Produces higher quality, more natural text-free images

Abstract

To prevent unauthorized use of text in images, Scene Text Removal (STR) has become a crucial task. It focuses on automatically removing text and replacing it with a natural, text-less background while preserving significant details such as texture, color, and contrast. Despite its importance in privacy protection, STR faces several challenges, including boundary artifacts, inconsistent texture and color, and preserving correct shadows. Most STR approaches estimate a text region mask to train a model, solving for image translation or inpainting to generate a text-free image. Thus, the quality of the generated image depends on the accuracy of the inpainting mask and the generator's capability. In this work, we leverage the superior capabilities of diffusion models in generating high-quality, consistent images to address the STR problem. We introduce a ControlNet diffusion model, treating STR as an inpainting task. To enhance the model's robustness, we develop a mask pretraining pipeline to condition our diffusion model. This involves training a masked autoencoder (MAE) using a combination of box masks and coarse stroke masks, and fine-tuning it using masks derived from our novel segmentation-based mask refinement framework. This framework iteratively refines an initial mask and segments it using the SLIC and Hierarchical Feature Selection (HFS) algorithms to produce an accurate final text mask. This improves mask prediction and utilizes rich textural information in natural scene images to provide accurate inpainting masks. Experiments on the SCUT-EnsText and SCUT-Syn datasets demonstrate that our method significantly outperforms existing state-of-the-art techniques.