Knowledge-Aware Audio-Grounded Generative Slot Filling for Limited Annotated Data

TL;DR

This paper introduces KA2G, a novel framework for speech-based task-oriented dialogue slot filling that leverages audio grounding and external knowledge to improve data efficiency and robustness against ASR errors in few-shot and zero-shot scenarios.

Contribution

KA2G is a new audio-grounded generative approach that enhances slot filling in spoken dialogue systems by integrating multi-modal data and external knowledge, especially for low-resource settings.

Findings

KA2G outperforms prior methods on SLURP dataset.

Significant improvements in few-shot and zero-shot learning scenarios.

Grounding in audio modality increases robustness to ASR errors.

Abstract

Manually annotating fine-grained slot-value labels for task-oriented dialogue (ToD) systems is an expensive and time-consuming endeavour. This motivates research into slot-filling methods that operate with limited amounts of labelled data. Moreover, the majority of current work on ToD is based solely on text as the input modality, neglecting the additional challenges of imperfect automatic speech recognition (ASR) when working with spoken language. In this work, we propose a Knowledge-Aware Audio-Grounded generative slot-filling framework, termed KA2G, that focuses on few-shot and zero-shot slot filling for ToD with speech input. KA2G achieves robust and data-efficient slot filling for speech-based ToD by 1) framing it as a text generation task, 2) grounding text generation additionally in the audio modality, and 3) conditioning on available external knowledge (e.g. a predefined list of possible slot values). We show that combining both modalities within the KA2G framework improves the robustness against ASR errors. Further, the knowledge-aware slot-value generator in KA2G, implemented via a pointer generator mechanism, particularly benefits few-shot and zero-shot learning. Experiments, conducted on the standard speech-based single-turn SLURP dataset and a multi-turn dataset extracted from a commercial ToD system, display strong and consistent gains over prior work, especially in few-shot and zero-shot setups.