TripleFDS: Triple Feature Disentanglement and Synthesis for Scene Text Editing

TL;DR

TripleFDS introduces a novel scene text editing framework that disentangles text style, content, and background, enabling more flexible and accurate image modifications with state-of-the-art results.

Contribution

It proposes a new triple feature disentanglement framework and dataset, enhancing controllability and visual consistency in scene text editing tasks.

Findings

Achieves state-of-the-art SSIM of 44.54 and accuracy of 93.58% on benchmarks.

Supports new editing operations like style replacement and background transfer.

Trained on 125,000 SCB Groups for robust disentanglement.

Abstract

Scene Text Editing (STE) aims to naturally modify text in images while preserving visual consistency, the decisive factors of which can be divided into three parts, i.e., text style, text content, and background. Previous methods have struggled with incomplete disentanglement of editable attributes, typically addressing only one aspect - such as editing text content - thus limiting controllability and visual consistency. To overcome these limitations, we propose TripleFDS, a novel framework for STE with disentangled modular attributes, and an accompanying dataset called SCB Synthesis. SCB Synthesis provides robust training data for triple feature disentanglement by utilizing the "SCB Group", a novel construct that combines three attributes per image to generate diverse, disentangled training groups. Leveraging this construct as a basic training unit, TripleFDS first disentangles triple features, ensuring semantic accuracy through inter-group contrastive regularization and reducing redundancy through intra-sample multi-feature orthogonality. In the synthesis phase, TripleFDS performs feature remapping to prevent "shortcut" phenomena during reconstruction and mitigate potential feature leakage. Trained on 125,000 SCB Groups, TripleFDS achieves state-of-the-art image fidelity (SSIM of 44.54) and text accuracy (ACC of 93.58%) on the mainstream STE benchmarks. Besides superior performance, the more flexible editing of TripleFDS supports new operations such as style replacement and background transfer. Code: https://github.com/yusenbao01/TripleFDS