WHISMA: A Speech-LLM to Perform Zero-shot Spoken Language Understanding

TL;DR

WHISMA is a speech-LLM designed for zero-shot spoken language understanding, combining Whisper and Llama-3, achieving significant improvements in slot filling and domain generalization on SLU benchmarks.

Contribution

We introduce WHISMA, a novel speech-LLM that enhances zero-shot SLU performance and domain generalization through efficient fine-tuning and new benchmark evaluations.

Findings

26.6% relative improvement in zero-shot slot filling on SLURP

33.0% relative gain over Qwen-Audio on SLU-GLUE

Robust performance across diverse SLU tasks

Abstract

Speech large language models (speech-LLMs) integrate speech and text-based foundation models to provide a unified framework for handling a wide range of downstream tasks. In this paper, we introduce WHISMA, a speech-LLM tailored for spoken language understanding (SLU) that demonstrates robust performance in various zero-shot settings. WHISMA combines the speech encoder from Whisper with the Llama-3 LLM, and is fine-tuned in a parameter-efficient manner on a comprehensive collection of SLU-related datasets. Our experiments show that WHISMA significantly improves the zero-shot slot filling performance on the SLURP benchmark, achieving a relative gain of 26.6% compared to the current state-of-the-art model. Furthermore, to evaluate WHISMA's generalisation capabilities to unseen domains, we develop a new task-agnostic benchmark named SLU-GLUE. The evaluation results indicate that WHISMA outperforms an existing speech-LLM (Qwen-Audio) with a relative gain of 33.0%.