Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences

TL;DR

This paper explores modeling rakugo speech synthesis using advanced neural models to entertain audiences, highlighting current limitations and insights for future improvements in character distinguishability and expressiveness.

Contribution

It applies Tacotron 2 and enhancements to rakugo speech synthesis, providing new insights into entertainment-focused speech quality beyond naturalness.

Findings

Synthesized speech lacks professional-level quality.

Character distinguishability and content understandability are crucial for entertainment.

Richer fundamental frequency expression enhances entertainment value.

Abstract

We have been investigating rakugo speech synthesis as a challenging example of speech synthesis that entertains audiences. Rakugo is a traditional Japanese form of verbal entertainment similar to a combination of one-person stand-up comedy and comic storytelling and is popular even today. In rakugo, a performer plays multiple characters, and conversations or dialogues between the characters make the story progress. To investigate how close the quality of synthesized rakugo speech can approach that of professionals' speech, we modeled rakugo speech using Tacotron 2, a state-of-the-art speech synthesis system that can produce speech that sounds as natural as human speech albeit under limited conditions, and an enhanced version of it with self-attention to better consider long-term dependencies. We also used global style tokens and manually labeled context features to enrich speaking styles. Through a listening test, we measured not only naturalness but also distinguishability of characters, understandability of the content, and the degree of entertainment. Although we found that the speech synthesis models could not yet reach the professional level, the results of the listening test provided interesting insights: 1) we should not focus only on the naturalness of synthesized speech but also the distinguishability of characters and the understandability of the content to further entertain audiences; 2) the fundamental frequency (fo) expressions of synthesized speech are poorer than those of human speech, and more entertaining speech should have richer fo expression. Although there is room for improvement, we believe this is an important stepping stone toward achieving entertaining speech synthesis at the professional level.