Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue

TL;DR

This paper evaluates speech large language models' ability to identify and understand speakers in spoken dialogue, revealing they mainly rely on conversation content rather than speaker identity, and proposes better evaluation tasks.

Contribution

The study highlights the limited speaker awareness of current SpeechLLMs and introduces a new dataset and evaluation framework focused on speaker identity recognition.

Findings

SpeechLLMs perform well on context-based questions without speaker identification.

Models show limited ability to recognize speaker identities from audio.

Proposed tasks better evaluate speaker recognition capabilities in SpeechLLMs.

Abstract

In recent years, we have observed a rapid advancement in speech language models (SpeechLLMs), catching up with humans' listening and reasoning abilities. SpeechLLMs have demonstrated impressive spoken dialog question-answering (SQA) performance in benchmarks like Gaokao, the English listening test of the college entrance exam in China, which seemingly requires understanding both the spoken content and voice characteristics of speakers in a conversation. However, after carefully examining Gaokao's questions, we find the correct answers to many questions can be inferred from the conversation transcript alone, i.e.\ without speaker segmentation and identification. Our evaluation of state-of-the-art models Qwen-Audio and WavLLM on both Gaokao and our proposed "What Do You Like?" dataset shows a significantly higher accuracy in these context-based questions than in identity-critical questions, which can only be answered reliably with correct speaker identification. The results and analysis suggest that when solving SQA, the current SpeechLLMs exhibit limited speaker awareness from the audio and behave similarly to an LLM reasoning from the conversation transcription without sound. We propose that tasks focused on identity-critical questions could offer a more accurate evaluation framework of SpeechLLMs in SQA.