OpenACE: An Open Benchmark for Evaluating Audio Coding Performance

TL;DR

This paper introduces OpenACE, an open-source benchmark for evaluating audio and speech coding quality across diverse content types, enabling fairer and more reproducible comparisons of different codecs.

Contribution

It provides a comprehensive, open, and standardized benchmark for audio coding evaluation, including traditional and machine learning-based codecs, with diverse content and open test vectors.

Findings

OpenACE enables fairer comparisons of codecs on diverse data.

The benchmark includes evaluations of Opus, EVS, and LC3+ codecs.

Quality variations in emotional speech encoding at 16 kbps are demonstrated.

Abstract

Audio and speech coding lack unified evaluation and open-source testing. Many candidate systems were evaluated on proprietary, non-reproducible, or small data, and machine learning-based codecs are often tested on datasets with similar distributions as trained on, which is unfairly compared to digital signal processing-based codecs that usually work well with unseen data. This paper presents a full-band audio and speech coding quality benchmark with more variable content types, including traditional open test vectors. An example use case of audio coding quality assessment is presented with open-source Opus, 3GPP's EVS, and recent ETSI's LC3 with LC3+ used in Bluetooth LE Audio profiles. Besides, quality variations of emotional speech encoding at 16 kbps are shown. The proposed open-source benchmark contributes to audio and speech coding democratization and is available at https://github.com/JozefColdenhoff/OpenACE.