PAREDA: A Multi-Accent Speech Dataset of Natural Language Processing Research Discussions

TL;DR

PAREDA is a novel multi-accent speech dataset featuring natural language processing research discussions, designed to evaluate and improve the robustness of ASR systems across diverse accents and spontaneous speech.

Contribution

Introduces PAREDA, the first multi-accent dataset of NLP discussion speech, and demonstrates its utility in evaluating and enhancing ASR model performance on real-world, accented speech.

Findings

State-of-the-art models perform worse on PAREDA without fine-tuning.

Fine-tuning on PAREDA significantly reduces Word Error Rate.

PAREDA captures linguistic features often missing from existing datasets.

Abstract

While modern Automatic Speech Recognition (ASR) systems achieve high accuracy on benchmark corpora, their performance often degrades when there is real-world variability. This work focuses on variability arising due to accented, spontaneous, and domain-specific speech. In particular, we introduce PAper REading DAtaset (PAREDA), a first-of-its-kind multi-accent speech dataset consisting of discussions on academic Natural Language Processing (NLP) papers between speakers with Australian, Indian-English, and Chinese English accents. Each session elicits a spontaneous monologue (a summary of a paper's abstract) and a non-monologue (a question-and-answer session between participants), resulting in a corpus rich with technical jargon and conversational phenomena. We evaluate the performance of SOTA ASR models on PAREDA, analysing the impact of accent mixing and increased speech rate. Our results show that, in the zero-shot setting, models perform worse, confirming the dataset's challenging nature. However, fine-tuning on PAREDA significantly reduces the Word Error Rate (WER), demonstrating that our dataset captures linguistic characteristics often missing from existing corpora. PAREDA serves as a valuable new resource for building and evaluating more robust and inclusive ASR systems for specialised, real-world applications.