PolySinger: Singing-Voice to Singing-Voice Translation from English to Japanese

TL;DR

PolySinger is the first system designed for singing-voice to singing-voice translation from English to Japanese, aiming to bridge the gap between speech and singing translation with a controlled, cascaded approach.

Contribution

It introduces PolySinger, the first SV2SVT system for lyrics translation from English to Japanese, and proposes a cascaded framework for improved control and potential in singing translation.

Findings

PolySinger achieves promising translation quality based on MOS scores.

Native Japanese speakers' feedback highlights strengths and areas for improvement.

The study establishes a foundation for future research in singing-voice translation.

Abstract

The speech domain prevails in the spotlight for several natural language processing (NLP) tasks while the singing domain remains less explored. The culmination of NLP is the speech-to-speech translation (S2ST) task, referring to translation and synthesis of human speech. A disparity between S2ST and the possible adaptation to the singing domain, which we describe as singing-voice to singing-voice translation (SV2SVT), is becoming prominent as the former is progressing ever faster, while the latter is at a standstill. Singing-voice synthesis systems are overcoming the barrier of multi-lingual synthesis, despite limited attention has been paid to multi-lingual songwriting and song translation. This paper endeavors to determine what is required for successful SV2SVT and proposes PolySinger (Polyglot Singer): the first system for SV2SVT, performing lyrics translation from English to Japanese. A cascaded approach is proposed to establish a framework with a high degree of control which can potentially diminish the disparity between SV2SVT and S2ST. The performance of PolySinger is evaluated by a mean opinion score test with native Japanese speakers. Results and in-depth discussions with test subjects suggest a solid foundation for SV2SVT, but several shortcomings must be overcome, which are discussed for the future of SV2SVT.