SLURP: A Spoken Language Understanding Resource Package

TL;DR

SLURP is a comprehensive resource package for spoken language understanding, including a large diverse dataset, baseline models, and a new evaluation metric to advance research in audio-based semantic understanding.

Contribution

It introduces a new challenging English SLU dataset, competitive baselines, and a transparent metric for detailed error analysis, addressing limitations of existing resources.

Findings

The dataset covers 18 diverse domains, surpassing existing datasets in size and diversity.

Baseline models demonstrate competitive performance on the new dataset.

The new metric enables detailed error analysis for entity labelling improvements.

Abstract

Spoken Language Understanding infers semantic meaning directly from audio data, and thus promises to reduce error propagation and misunderstandings in end-user applications. However, publicly available SLU resources are limited. In this paper, we release SLURP, a new SLU package containing the following: (1) A new challenging dataset in English spanning 18 domains, which is substantially bigger and linguistically more diverse than existing datasets; (2) Competitive baselines based on state-of-the-art NLU and ASR systems; (3) A new transparent metric for entity labelling which enables a detailed error analysis for identifying potential areas of improvement. SLURP is available at https: //github.com/pswietojanski/slurp.