ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoning

TL;DR

ViDove is a multimodal translation system that uses visual context and memory modules to improve translation quality, especially for complex and long-form content, outperforming previous models.

Contribution

The paper introduces ViDove, a novel multimodal translation agent with memory-augmented reasoning, and presents DoveBench, a new benchmark for long-form video translation.

Findings

28% BLEU score improvement over baselines

15% SubER improvement in translation quality

Effective use of visual context and memory modules

Abstract

LLM-based translation agents have achieved highly human-like translation results and are capable of handling longer and more complex contexts with greater efficiency. However, they are typically limited to text-only inputs. In this paper, we introduce ViDove, a translation agent system designed for multimodal input. Inspired by the workflow of human translators, ViDove leverages visual and contextual background information to enhance the translation process. Additionally, we integrate a multimodal memory system and long-short term memory modules enriched with domain-specific knowledge, enabling the agent to perform more accurately and adaptively in real-world scenarios. As a result, ViDove achieves significantly higher translation quality in both subtitle generation and general translation tasks, with a 28% improvement in BLEU scores and a 15% improvement in SubER compared to previous state-of-the-art baselines. Moreover, we introduce DoveBench, a new benchmark for long-form automatic video subtitling and translation, featuring 17 hours of high-quality, human-annotated data. Our code is available here: https://github.com/pigeonai-org/ViDove