Automatic Song Translation for Tonal Languages

TL;DR

This paper introduces GagaST, an unsupervised system for translating songs into tonal languages like Mandarin, effectively balancing meaning, singability, and intelligibility through novel metrics and a new benchmark.

Contribution

It presents the first unsupervised approach for English-Mandarin song translation that aligns tones with melody while preserving meaning and singability.

Findings

GagaST outperforms baseline models in balancing semantics and singability.

The new benchmark enables standardized evaluation of song translation quality.

Both automatic and human evaluations confirm GagaST's effectiveness.

Abstract

This paper develops automatic song translation (AST) for tonal languages and addresses the unique challenge of aligning words' tones with melody of a song in addition to conveying the original meaning. We propose three criteria for effective AST -- preserving meaning, singability and intelligibility -- and design metrics for these criteria. We develop a new benchmark for English--Mandarin song translation and develop an unsupervised AST system, Guided AliGnment for Automatic Song Translation (GagaST), which combines pre-training with three decoding constraints. Both automatic and human evaluations show GagaST successfully balances semantics and singability.