Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation

TL;DR

This paper introduces M4Doc, a framework that aligns image-only encoders with multimodal large language models to improve document image machine translation, achieving better generalization and efficiency.

Contribution

M4Doc is the first to align single-modality encoders with multimodal models for DIMT, enhancing translation quality and generalization with a lightweight approach.

Findings

Significant improvements in translation quality across domains.

Enhanced cross-domain generalization in document scenarios.

Maintains computational efficiency during inference.

Abstract

Document Image Machine Translation (DIMT) aims to translate text within document images, facing generalization challenges due to limited training data and the complex interplay between visual and textual information. To address these challenges, we introduce M4Doc, a novel single-to-mix modality alignment framework leveraging Multimodal Large Language Models (MLLMs). M4Doc aligns an image-only encoder with the multimodal representations of an MLLM, pre-trained on large-scale document image datasets. This alignment enables a lightweight DIMT model to learn crucial visual-textual correlations during training. During inference, M4Doc bypasses the MLLM, maintaining computational efficiency while benefiting from its multimodal knowledge. Comprehensive experiments demonstrate substantial improvements in translation quality, especially in cross-domain generalization and challenging document image scenarios.