Massive Sound Embedding Benchmark (MSEB)

TL;DR

The Massive Sound Embedding Benchmark (MSEB) provides a comprehensive, extensible framework for evaluating auditory capabilities in multimodal systems through diverse tasks and datasets, including a new large-scale voice dataset.

Contribution

MSEB introduces a new benchmark framework with multiple core tasks and datasets for assessing sound embeddings in multimodal AI systems, including a novel large-scale voice dataset.

Findings

Initial experiments reveal significant performance headroom.

Diverse datasets enable comprehensive evaluation.

Framework encourages community contributions.

Abstract

Audio is a critical component of multimodal perception, and any truly intelligent system must demonstrate a wide range of auditory capabilities. These capabilities include transcription, classification, retrieval, reasoning, segmentation, clustering, reranking, and reconstruction. Fundamentally, each task involves transforming a raw audio signal into a meaningful 'embedding' - be it a single vector, a sequence of continuous or discrete representations, or another structured form - which then serves as the basis for generating the task's final response. To accelerate progress towards robust machine auditory intelligence, we present the Massive Sound Embedding Benchmark (MSEB): an extensible framework designed to evaluate the auditory components of any multimodal system. In its first release, MSEB offers a comprehensive suite of eight core tasks, with more planned for the future, supported by diverse datasets, including the new, large-scale Simple Voice Questions (SVQ) dataset. Our initial experiments establish clear performance headrooms, highlighting the significant opportunity to improve real-world multimodal experiences where audio is a core signal. We encourage the research community to use MSEB to assess their algorithms and contribute to its growth. The library is publicly hosted at github.