Ensuring Consistency for In-Image Translation

TL;DR

This paper introduces HCIIT, a two-stage framework that enhances in-image translation by ensuring translation and style consistency through multimodal models and diffusion techniques, resulting in more coherent and high-quality translated images.

Contribution

The paper presents a novel two-stage approach combining multimodal language models and diffusion models to maintain consistency in in-image translation tasks, addressing a key gap in existing methods.

Findings

Effective in maintaining translation and style consistency.

Produces high-quality, style-coherent translated images.

Validated on both curated and real-world datasets.

Abstract

The in-image machine translation task involves translating text embedded within images, with the translated results presented in image format. While this task has numerous applications in various scenarios such as film poster translation and everyday scene image translation, existing methods frequently neglect the aspect of consistency throughout this process. We propose the need to uphold two types of consistency in this task: translation consistency and image generation consistency. The former entails incorporating image information during translation, while the latter involves maintaining consistency between the style of the text-image and the original image, ensuring background integrity. To address these consistency requirements, we introduce a novel two-stage framework named HCIIT (High-Consistency In-Image Translation) which involves text-image translation using a multimodal multilingual large language model in the first stage and image backfilling with a diffusion model in the second stage. Chain of thought learning is utilized in the first stage to enhance the model's ability to leverage image information during translation. Subsequently, a diffusion model trained for style-consistent text-image generation ensures uniformity in text style within images and preserves background details. A dataset comprising 400,000 style-consistent pseudo text-image pairs is curated for model training. Results obtained on both curated test sets and authentic image test sets validate the effectiveness of our framework in ensuring consistency and producing high-quality translated images.