HPSU: A Benchmark for Human-Level Perception in Real-World Spoken Speech Understanding

TL;DR

HPSU introduces a comprehensive benchmark with over 20,000 samples to evaluate speech models' ability to understand latent intentions and emotions, revealing current models lag behind human perception.

Contribution

The paper presents HPSU, a new benchmark for assessing human-level perception in speech understanding, including a semi-automatic annotation process for complex, real-world data.

Findings

Models underperform compared to humans in understanding implicit emotions.

HPSU covers diverse tasks from speaker attributes to complex inferences.

Semi-automatic annotation improves data quality and efficiency.

Abstract

Recent advances in Speech Large Language Models (Speech LLMs) have led to great progress in speech understanding tasks such as Automatic Speech Recognition (ASR) and Speech Emotion Recognition (SER). However, whether these models can achieve human-level auditory perception, particularly in terms of their ability to comprehend latent intentions and implicit emotions in real-world spoken language, remains underexplored. To this end, we introduce the Human-level Perception in Spoken Speech Understanding (HPSU), a new benchmark for fully evaluating the human-level perceptual and understanding capabilities of Speech LLMs. HPSU comprises over 20,000 expert-validated spoken language understanding samples in English and Chinese. It establishes a comprehensive evaluation framework by encompassing a spectrum of tasks, ranging from basic speaker attribute recognition to complex inference of latent intentions and implicit emotions. To address the issues of data scarcity and high cost of manual annotation in real-world scenarios, we developed a semi-automatic annotation process. This process fuses audio, textual, and visual information to enable precise speech understanding and labeling, thus enhancing both annotation efficiency and quality. We systematically evaluate various open-source and proprietary Speech LLMs. The results demonstrate that even top-performing models still fall considerably short of human capabilities in understanding genuine spoken interactions. Consequently, HPSU will be useful for guiding the development of Speech LLMs toward human-level perception and cognition.