MAVL: A Multilingual Audio-Video Lyrics Dataset for Animated Song Translation

TL;DR

This paper introduces MAVL, a comprehensive multilingual multimodal dataset for animated song translation, and proposes SylAVL-CoT, a model leveraging audio, video, and syllabic constraints to improve translation quality.

Contribution

The paper presents the first multilingual multimodal benchmark for animated song translation and a novel syllable-constrained model that enhances singability and contextual accuracy.

Findings

SylAVL-CoT outperforms text-only models in singability.

Multimodal approaches improve translation quality.

MAVL enables richer, more expressive lyrics translation.

Abstract

Lyrics translation requires both accurate semantic transfer and preservation of musical rhythm, syllabic structure, and poetic style. In animated musicals, the challenge intensifies due to alignment with visual and auditory cues. We introduce Multilingual Audio-Video Lyrics Benchmark for Animated Song Translation (MAVL), the first multilingual, multimodal benchmark for singable lyrics translation. By integrating text, audio, and video, MAVL enables richer and more expressive translations than text-only approaches. Building on this, we propose Syllable-Constrained Audio-Video LLM with Chain-of-Thought SylAVL-CoT, which leverages audio-video cues and enforces syllabic constraints to produce natural-sounding lyrics. Experimental results demonstrate that SylAVL-CoT significantly outperforms text-based models in singability and contextual accuracy, emphasizing the value of multimodal, multilingual approaches for lyrics translation.