Advancing Topic Segmentation of Broadcasted Speech with Multilingual Semantic Embeddings

TL;DR

This paper introduces an end-to-end multilingual speech-based topic segmentation method that bypasses traditional transcript-based approaches, demonstrating competitive results on a new diverse broadcasted news dataset across six European languages and Hindi.

Contribution

The paper presents a novel end-to-end speech segmentation approach using multilingual semantic embeddings and establishes a new benchmark dataset for spoken news topic segmentation.

Findings

End-to-end model achieves a $P_k$ score of 0.2564 for English.

Multilingual training improves scores to 0.1988 and 0.2370.

Model and data scripts are publicly released for research use.

Abstract

Recent advancements in speech-based topic segmentation have highlighted the potential of pretrained speech encoders to capture semantic representations directly from speech. Traditionally, topic segmentation has relied on a pipeline approach in which transcripts of the automatic speech recognition systems are generated, followed by text-based segmentation algorithms. In this paper, we introduce an end-to-end scheme that bypasses this conventional two-step process by directly employing semantic speech encoders for segmentation. Focused on the broadcasted news domain, which poses unique challenges due to the diversity of speakers and topics within single recordings, we address the challenge of accessing topic change points efficiently in an end-to-end manner. Furthermore, we propose a new benchmark for spoken news topic segmentation by utilizing a dataset featuring approximately 1000 hours of publicly available recordings across six European languages and including an evaluation set in Hindi to test the model's cross-domain performance in a cross-lingual, zero-shot scenario. This setup reflects real-world diversity and the need for models adapting to various linguistic settings. Our results demonstrate that while the traditional pipeline approach achieves a state-of-the-art $P_k$ score of 0.2431 for English, our end-to-end model delivers a competitive $P_k$ score of 0.2564. When trained multilingually, these scores further improve to 0.1988 and 0.2370, respectively. To support further research, we release our model along with data preparation scripts, facilitating open research on multilingual spoken news topic segmentation.