SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks

TL;DR

This paper introduces a new suite of diverse spoken language understanding benchmark tasks based on freely available data, filling gaps in existing SLU evaluation resources and enabling better model development and comparison.

Contribution

It presents four novel SLU tasks, provides annotated datasets and baseline models, and analyzes the impact of speech recognition accuracy on SLU performance.

Findings

Baseline models demonstrate competitive performance on new tasks.

Speech recognition accuracy significantly affects SLU pipeline performance.

Annotated datasets and benchmarks are publicly available for research use.

Abstract

Spoken language understanding (SLU) tasks have been studied for many decades in the speech research community, but have not received as much attention as lower-level tasks like speech and speaker recognition. In particular, there are not nearly as many SLU task benchmarks, and many of the existing ones use data that is not freely available to all researchers. Recent work has begun to introduce such benchmark datasets for several tasks. In this work, we introduce several new annotated SLU benchmark tasks based on freely available speech data, which complement existing benchmarks and address gaps in the SLU evaluation landscape. We contribute four tasks: question answering and summarization involve inference over longer speech sequences; named entity localization addresses the speech-specific task of locating the targeted content in the signal; dialog act classification identifies the function of a given speech utterance. We follow the blueprint of the Spoken Language Understanding Evaluation (SLUE) benchmark suite. In order to facilitate the development of SLU models that leverage the success of pre-trained speech representations, we will be publishing for each task (i) annotations for a relatively small fine-tuning set, (ii) annotated development and test sets, and (iii) baseline models for easy reproducibility and comparisons. In this work, we present the details of data collection and annotation and the performance of the baseline models. We also perform sensitivity analysis of pipeline models' performance (speech recognizer + text model) to the speech recognition accuracy, using more than 20 state-of-the-art speech recognition models.