Prompting Whisper for QA-driven Zero-shot End-to-end Spoken Language Understanding

TL;DR

This paper introduces a zero-shot end-to-end spoken language understanding system using Whisper, leveraging a QA framework and prefix-tuning to achieve high accuracy with fewer parameters, outperforming recent benchmarks.

Contribution

It presents a novel approach combining Whisper with a QA framework and prefix-tuning for efficient zero-shot SLU, reducing model complexity while maintaining high performance.

Findings

40.7% absolute gain in slot filling (SLU-F1) on SLURP

Performs comparably to Whisper-GPT-2 system

34.8% reduction in model parameters

Abstract

Zero-shot spoken language understanding (SLU) enables systems to comprehend user utterances in new domains without prior exposure to training data. Recent studies often rely on large language models (LLMs), leading to excessive footprints and complexity. This paper proposes the use of Whisper, a standalone speech processing model, for zero-shot end-to-end (E2E) SLU. To handle unseen semantic labels, SLU tasks are integrated into a question-answering (QA) framework, which prompts the Whisper decoder for semantics deduction. The system is efficiently trained with prefix-tuning, optimising a minimal set of parameters rather than the entire Whisper model. We show that the proposed system achieves a 40.7% absolute gain for slot filling (SLU-F1) on SLURP compared to a recently introduced zero-shot benchmark. Furthermore, it performs comparably to a Whisper-GPT-2 modular system under both in-corpus and cross-corpus evaluation settings, but with a relative 34.8% reduction in model parameters.