SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases

TL;DR

This paper introduces SCENEBench, a comprehensive benchmark for evaluating large audio language models across diverse real-world audio understanding tasks, highlighting current model gaps and guiding future improvements.

Contribution

The paper presents SCENEBench, a novel benchmark suite for assessing broad audio comprehension in LALMs, covering background sounds, noise localization, cross-lingual speech, and vocal recognition.

Findings

Model performance varies significantly across tasks.

Some models perform below random chance on certain tasks.

Benchmark results highlight specific areas for model improvement.

Abstract

Advances in large language models (LLMs) have enabled significant capabilities in audio processing, resulting in state-of-the-art models now known as Large Audio Language Models (LALMs). However, minimal work has been done to measure audio understanding beyond automatic speech recognition (ASR). This paper closes that gap by proposing a benchmark suite, SCENEBench (Spatial, Cross-lingual, Environmental, Non-speech Evaluation), that targets a broad form of audio comprehension across four real-world categories: background sound understanding, noise localization, cross-linguistic speech understanding, and vocal characterizer recognition. These four categories are selected based on understudied needs from accessibility technology and industrial noise monitoring. In addition to performance, we also measure model latency. The purpose of this benchmark suite is to assess audio beyond just what words are said - rather, how they are said and the non-speech components of the audio. Because our audio samples are synthetically constructed (e.g., by overlaying two natural audio samples), we further validate our benchmark against 20 natural audio items per task, sub-sampled from existing datasets to match our task criteria, to assess ecological validity. We assess five state-of-the-art LALMs and find critical gaps: performance varies across tasks, with some tasks performing below random chance and others achieving high accuracy. These results provide direction for targeted improvements in model capabilities.