ProsAudit, a prosodic benchmark for self-supervised speech models

TL;DR

ProsAudit is a new benchmark for evaluating prosodic knowledge in self-supervised speech models, focusing on prosodic boundaries and pause distinctions, with insights into model performance related to language and data size.

Contribution

Introduces ProsAudit, a comprehensive benchmark for prosodic understanding in SSL speech models, including evaluation metrics and human comparison scores.

Findings

Models perform above chance on prosodic tasks

Native models outperform non-native ones in lexical task

Larger models trained on more data perform better

Abstract

We present ProsAudit, a benchmark in English to assess structural prosodic knowledge in self-supervised learning (SSL) speech models. It consists of two subtasks, their corresponding metrics, and an evaluation dataset. In the protosyntax task, the model must correctly identify strong versus weak prosodic boundaries. In the lexical task, the model needs to correctly distinguish between pauses inserted between words and within words. We also provide human evaluation scores on this benchmark. We evaluated a series of SSL models and found that they were all able to perform above chance on both tasks, even when evaluated on an unseen language. However, non-native models performed significantly worse than native ones on the lexical task, highlighting the importance of lexical knowledge in this task. We also found a clear effect of size with models trained on more data performing better in the two subtasks.