WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wild

TL;DR

WildSpeech-Bench introduces a comprehensive evaluation benchmark for end-to-end speech LLMs, addressing speech-specific challenges and enabling more accurate assessment of model performance in real-world spoken scenarios.

Contribution

It is the first benchmark specifically designed to evaluate end-to-end speech LLMs with speech-specific phenomena and a query-aware evaluation method.

Findings

Significant performance differences across speech models.

Enhanced evaluation accuracy with query-aware methods.

Diverse real-world speech data improves assessment.

Abstract

Recent multi-modal Large Language Models (LLMs) such as GPT-4o have demonstrated strong capabilities of direct speech interaction. However, the lack of specialized and comprehensive benchmarks for end-to-end speech LLM evaluation hinders optimizing the user experience of Audio LLMs in real-world applications. Existing evaluation methods often adapt text-based benchmarks, overlooking speech's unique characteristics and challenges, including prosody, homophones, stuttering, and differing user expectations. Here, we introduce the first comprehensive benchmark designed to systematically evaluate end-to-end speechLLMs in practical speech conversations. We systematically curate real-world chat data relevant to spoken scenarios, introduce diversity in speaker attributes and acoustic conditions, and augment the dataset with speech-specific phenomena. We further design a query-aware evaluation method to use customized evaluation checklists and prompts to enhance the accuracy of automatic evaluation. We conduct comprehensive testing and detailed analysis of various mainstream speech models, revealing significant differences in model performance across different speech scenarios. The use of query-aware evaluation further enables a finer-grained assessment under various speech-specific scenarios. Our benchmark can provide valuable insights for speech model development and evaluation.