Do What I Say: A Spoken Prompt Dataset for Instruction-Following

TL;DR

The paper introduces DOWIS, a multilingual spoken prompt dataset for evaluating speech large language models in realistic, speech-based interaction scenarios across multiple tasks and languages.

Contribution

It provides a comprehensive multilingual spoken prompt dataset and benchmarks SLLMs, revealing the impact of prompt modality, style, and language on performance.

Findings

Text prompts outperform spoken prompts overall.

Spoken prompts are more effective for speech output tasks.

Low-resource and cross-lingual settings favor text prompts.

Abstract

Speech Large Language Models (SLLMs) have rapidly expanded, supporting a wide range of tasks. These models are typically evaluated using text prompts, which may not reflect real-world scenarios where users interact with speech. To address this gap, we introduce DoWhatISay (DOWIS), a multilingual dataset of human-recorded spoken and written prompts designed to pair with any existing benchmark for realistic evaluation of SLLMs under spoken instruction conditions. Spanning 9 tasks and 11 languages, it provides 10 prompt variants per task-language pair, across five styles. Using DOWIS, we benchmark state-of-the-art SLLMs, analyzing the interplay between prompt modality, style, language, and task type. Results show that text prompts consistently outperform spoken prompts, particularly for low-resource and cross-lingual settings. Only for tasks with speech output, spoken prompts do close the gap, highlighting the need for speech-based prompting in SLLM evaluation.