Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metrics

TL;DR

This paper evaluates spoken language models for conversational ASR, introducing new metrics to better assess semantic accuracy and robustness in multi-speaker scenarios, revealing strengths and weaknesses of LLM-based and modular approaches.

Contribution

It introduces tcpSemER, a semantic-aware metric, and provides a systematic comparison of LLM-based and modular ASR systems across challenging multi-speaker conditions.

Findings

LLM-based systems perform well with two speakers but degrade with more overlap.

Modular pipelines are more robust to increasing speaker count and overlap.

tcpSemER captures meaning errors missed by traditional metrics.

Abstract

Conversational automatic speech recognition remains challenging due to overlapping speech, far-field noise, and varying speaker counts. While recent LLM-based systems perform well on single-speaker benchmarks, their robustness in multi-speaker settings is unclear. We systematically compare LLM-based and modular pipeline approaches along four axes: overlap robustness, semantic fidelity, speaker count, and single- versus multi-channel input. To capture meaning-altering errors that conventional metrics miss, we introduce tcpSemER, which extends tcpWER by replacing Levenshtein distance with embedding-based semantic similarity. We further decompose tcpWER into overlapping and non-overlapping components for finer-grained analysis. Experiments across three datasets show that LLM-based systems are competitive in two-speaker settings but degrade as speaker count and overlap increase, whereas modular pipelines remain more robust.