Context-Informed Machine Translation of Manga using Multimodal Large Language Models

TL;DR

This paper explores the use of multimodal large language models for manga translation, incorporating visual elements to improve quality, and introduces a new dataset and software tools for benchmarking in this domain.

Contribution

It proposes a novel methodology leveraging multimodal LLMs for manga translation, introduces a Japanese-Polish translation dataset, and provides open-source benchmarking tools.

Findings

Achieves state-of-the-art results for Japanese-English manga translation.

Sets a new standard for Japanese-Polish translation.

Demonstrates the effectiveness of visual context in translation quality.

Abstract

Due to the significant time and effort required for handcrafting translations, most manga never leave the domestic Japanese market. Automatic manga translation is a promising potential solution. However, it is a budding and underdeveloped field and presents complexities even greater than those found in standard translation due to the need to effectively incorporate visual elements into the translation process to resolve ambiguities. In this work, we investigate to what extent multimodal large language models (LLMs) can provide effective manga translation, thereby assisting manga authors and publishers in reaching wider audiences. Specifically, we propose a methodology that leverages the vision component of multimodal LLMs to improve translation quality and evaluate the impact of translation unit size, context length, and propose a token efficient approach for manga translation. Moreover, we introduce a new evaluation dataset -- the first parallel Japanese-Polish manga translation dataset -- as part of a benchmark to be used in future research. Finally, we contribute an open-source software suite, enabling others to benchmark LLMs for manga translation. Our findings demonstrate that our proposed methods achieve state-of-the-art results for Japanese-English translation and set a new standard for Japanese-Polish.