Acoustic characterization of speech rhythm: going beyond metrics with recurrent neural networks

TL;DR

This study uses recurrent neural networks to analyze speech rhythm, revealing complex acoustic patterns that align with linguistic rhythmic typologies and surpass traditional metrics in capturing rhythm complexity.

Contribution

It demonstrates that deep learning models can effectively characterize speech rhythm and relate neural representations to established rhythmic metrics, advancing the understanding of speech prosody.

Findings

Neural network identified language with 40% accuracy on 10-second recordings.

Representations reflect speech rhythm typologies beyond simple clusters.

Correlations found between network activations and known rhythm metrics.

Abstract

Languages have long been described according to their perceived rhythmic attributes. The associated typologies are of interest in psycholinguistics as they partly predict newborns' abilities to discriminate between languages and provide insights into how adult listeners process non-native languages. Despite the relative success of rhythm metrics in supporting the existence of linguistic rhythmic classes, quantitative studies have yet to capture the full complexity of temporal regularities associated with speech rhythm. We argue that deep learning offers a powerful pattern-recognition approach to advance the characterization of the acoustic bases of speech rhythm. To explore this hypothesis, we trained a medium-sized recurrent neural network on a language identification task over a large database of speech recordings in 21 languages. The network had access to the amplitude envelopes and a variable identifying the voiced segments, assuming that this signal would poorly convey phonetic information but preserve prosodic features. The network was able to identify the language of 10-second recordings in 40% of the cases, and the language was in the top-3 guesses in two-thirds of the cases. Visualization methods show that representations built from the network activations are consistent with speech rhythm typologies, although the resulting maps are more complex than two separated clusters between stress and syllable-timed languages. We further analyzed the model by identifying correlations between network activations and known speech rhythm metrics. The findings illustrate the potential of deep learning tools to advance our understanding of speech rhythm through the identification and exploration of linguistically relevant acoustic feature spaces.