# CodecBench: A Comprehensive Benchmark for Acoustic and Semantic Evaluation

## TL;DR

CodecBench is a new comprehensive benchmark dataset designed to evaluate audio codecs' acoustic and semantic performance across complex, real-world scenarios, addressing limitations of previous simplistic evaluation methods.

## Contribution

It introduces a detailed evaluation dataset for audio codecs that captures complex acoustic and semantic information in diverse scenarios, enabling more accurate assessment.

## Key findings

- Identifies limitations of existing audio codec evaluations.
- Provides a dataset covering multiple data domains and complex scenarios.
- Facilitates future research to improve audio codec performance.

## Abstract

With the rise of multimodal large language models (LLMs), audio codec plays an increasingly vital role in encoding audio into discrete tokens, enabling integration of audio into text-based LLMs. Current audio codec captures two types of information: acoustic and semantic. As audio codec is applied to diverse scenarios in speech language model , it needs to model increasingly complex information and adapt to varied contexts, such as scenarios with multiple speakers, background noise, or richer paralinguistic information. However, existing codec's own evaluation has been limited by simplistic metrics and scenarios, and existing benchmarks for audio codec are not designed for complex application scenarios, which limits the assessment performance on complex datasets for acoustic and semantic capabilities. We introduce CodecBench, a comprehensive evaluation dataset to assess audio codec performance from both acoustic and semantic perspectives across four data domains. Through this benchmark, we aim to identify current limitations, highlight future research directions, and foster advances in the development of audio codec. The codes are available at https://github.com/RayYuki/CodecBench.