Evaluating generative audio systems and their metrics

TL;DR

This paper critically evaluates current generative audio systems and metrics, revealing that existing objective measures do not reliably reflect perceptual audio quality, based on comparative analysis and listening tests.

Contribution

It provides a comprehensive comparison of state-of-the-art audio synthesis methods with objective metrics and human listening, highlighting the inadequacy of current evaluation metrics.

Findings

Objective metrics do not correlate well with perceptual quality.

Listening tests reveal discrepancies between metrics and human judgment.

Current evaluation methods are insufficient for assessing audio quality.

Abstract

Recent years have seen considerable advances in audio synthesis with deep generative models. However, the state-of-the-art is very difficult to quantify; different studies often use different evaluation methodologies and different metrics when reporting results, making a direct comparison to other systems difficult if not impossible. Furthermore, the perceptual relevance and meaning of the reported metrics in most cases unknown, prohibiting any conclusive insights with respect to practical usability and audio quality. This paper presents a study that investigates state-of-the-art approaches side-by-side with (i) a set of previously proposed objective metrics for audio reconstruction, and with (ii) a listening study. The results indicate that currently used objective metrics are insufficient to describe the perceptual quality of current systems.