TELEVAL: A Dynamic Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarios

TL;DR

TELEVAL is a new dynamic benchmark for evaluating Chinese spoken language models in realistic, user-centered interaction scenarios, focusing on content understanding and interactional appropriateness.

Contribution

It introduces TELEVAL, a benchmark that emphasizes natural, interactional aspects of spoken language models, addressing limitations of existing task-focused evaluation methods.

Findings

Current SLMs excel in content comprehension but lack natural interaction skills.

Models struggle with producing socially appropriate and colloquial responses.

TELEVAL highlights the gap between semantic accuracy and natural conversational behavior.

Abstract

Spoken language models (SLMs) have advanced rapidly in recent years, accompanied by a growing number of evaluation benchmarks. However, most existing benchmarks emphasize task completion and capability scaling, while remaining poorly aligned with how users interact with SLMs in real-world spoken conversations. Effective spoken interaction requires not only accurate understanding of user intent and content, but also the ability to respond with appropriate interactional strategies. In this paper, we present TELEVAL, a dynamic, user-centered benchmark for evaluating SLMs in realistic Chinese spoken interaction scenarios. TELEVAL consolidates evaluation into two core aspects. Reliable Content Fulfillment assesses whether models can comprehend spoken inputs and produce semantically correct responses. Interactional Appropriateness evaluates whether models act as socially capable interlocutors, requiring them not only to generate human-like, colloquial responses, but also to implicitly incorporate paralinguistic cues for natural interaction. Experiments reveal that, despite strong performance on semantic and knowledge-oriented tasks, current SLMs still struggle to produce natural and interactionally appropriate responses, highlighting the need for more interaction-faithful evaluation.