URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models

TL;DR

URO-Bench is a comprehensive evaluation benchmark for end-to-end spoken dialogue models, covering multilingualism, multi-round dialogues, and paralinguistics to advance speech-to-speech AI research.

Contribution

This paper introduces URO-Bench, the first S2S benchmark evaluating SDMs across multiple complex speech and dialogue capabilities, filling a significant evaluation gap.

Findings

Current SDMs perform well in daily QA tasks.

They lag behind LLMs in instruction-following and suffer from catastrophic forgetting.

Performance in paralinguistic and audio understanding is subpar.

Abstract

Recent advances in large language models (LLMs) have driven significant progress in end-to-end spoken dialogue models (SDMs). In contrast to text-based LLMs, the evaluation framework for SDMs should encompass both cognitive dimensions (e.g., logical reasoning, knowledge) and speech-related aspects (e.g., paralinguistic cues, audio quality). However, there is still a lack of comprehensive evaluations for SDMs in speech-to-speech (S2S) scenarios. To address this gap, we propose URO-Bench, an extensive benchmark for SDMs. Notably, URO-Bench is the first S2S benchmark that covers evaluations about multilingualism, multi-round dialogues, and paralinguistics. Our benchmark is divided into two difficulty levels: basic track and pro track, each comprising 20 test sets, evaluating the spoken dialogue model's abilities in Understanding, Reasoning, and Oral conversation. Evaluations on our proposed benchmark reveal that current open-source SDMs perform rather well in daily QA tasks, but lag behind their backbone LLMs in terms of instruction-following ability and also suffer from catastrophic forgetting. Their performance in advanced evaluations of paralinguistic information and audio understanding remains subpar, highlighting the need for further research in this direction. We hope that URO-Bench can facilitate the development of spoken dialogue models by providing a multifaceted evaluation of existing models and helping to track progress in this area.