Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation

TL;DR

Translatotron-V(ision) is an end-to-end in-image machine translation model that effectively translates images containing text into translated images, reducing parameters and outperforming previous pixel-level models.

Contribution

The paper introduces Translatotron-V(ision), a novel end-to-end IIMT model with a new architecture and training framework that improves translation quality and efficiency.

Findings

Achieves competitive performance with 70.9% of parameters of cascaded models.

Outperforms existing pixel-level end-to-end IIMT models.

Introduces Structure-BLEU, a new evaluation metric for image translation quality.

Abstract

In-image machine translation (IIMT) aims to translate an image containing texts in source language into an image containing translations in target language. In this regard, conventional cascaded methods suffer from issues such as error propagation, massive parameters, and difficulties in deployment and retaining visual characteristics of the input image. Thus, constructing end-to-end models has become an option, which, however, faces two main challenges: 1) the huge modeling burden, as it is required to simultaneously learn alignment across languages and preserve the visual characteristics of the input image; 2) the difficulties of directly predicting excessively lengthy pixel sequences. In this paper, we propose \textit{Translatotron-V(ision)}, an end-to-end IIMT model consisting of four modules. In addition to an image encoder, and an image decoder, our model contains a target text decoder and an image tokenizer. Among them, the target text decoder is used to alleviate the language alignment burden, and the image tokenizer converts long sequences of pixels into shorter sequences of visual tokens, preventing the model from focusing on low-level visual features. Besides, we present a two-stage training framework for our model to assist the model in learning alignment across modalities and languages. Finally, we propose a location-aware evaluation metric called Structure-BLEU to assess the translation quality of the generated images. Experimental results demonstrate that our model achieves competitive performance compared to cascaded models with only 70.9\% of parameters, and significantly outperforms the pixel-level end-to-end IIMT model.