Synchronising speech segments with musical beats in Mandarin and English singing

TL;DR

This study investigates how speech segments align with musical beats in Mandarin and English singing, emphasizing the importance of temporal relationship information for improving singing voice synthesis.

Contribution

It provides a detailed analysis of segment-beat synchronization in singing data, highlighting linguistic factors influencing beat placement across languages.

Findings

Beat presence depends more on segment duration than sonority.

Sonority hierarchy and P-centre theory relate closely to beat location.

Cross-linguistic variations observed between Mandarin and English.

Abstract

Generating synthesised singing voice with models trained on speech data has many advantages due to the models' flexibility and controllability. However, since the information about the temporal relationship between segments and beats are lacking in speech training data, the synthesised singing may sound off-beat at times. Therefore, the availability of the information on the temporal relationship between speech segments and music beats is crucial. The current study investigated the segment-beat synchronisation in singing data, with hypotheses formed based on the linguistics theories of P-centre and sonority hierarchy. A Mandarin corpus and an English corpus of professional singing data were manually annotated and analysed. The results showed that the presence of musical beats was more dependent on segment duration than sonority. However, the sonority hierarchy and the P-centre theory were highly related to the location of beats. Mandarin and English demonstrated cross-linguistic variations despite exhibiting common patterns.