Exploring In-Image Machine Translation with Real-World Background

TL;DR

This paper introduces a new dataset and a novel DebackX model for in-image machine translation in complex real-world backgrounds, significantly improving translation quality and visual coherence.

Contribution

The paper presents a real-world background dataset for IIMT and proposes DebackX, a model that separates, translates, and fuses text with backgrounds for better results.

Findings

DebackX outperforms previous models in translation quality.

The dataset enables more realistic IIMT research.

Experimental results show improved visual coherence.

Abstract

In-Image Machine Translation (IIMT) aims to translate texts within images from one language to another. Previous research on IIMT was primarily conducted on simplified scenarios such as images of one-line text with black font in white backgrounds, which is far from reality and impractical for applications in the real world. To make IIMT research practically valuable, it is essential to consider a complex scenario where the text backgrounds are derived from real-world images. To facilitate research of complex scenario IIMT, we design an IIMT dataset that includes subtitle text with real-world background. However previous IIMT models perform inadequately in complex scenarios. To address the issue, we propose the DebackX model, which separates the background and text-image from the source image, performs translation on text-image directly, and fuses the translated text-image with the background, to generate the target image. Experimental results show that our model achieves improvements in both translation quality and visual effect.