Do Neural Codecs Generalize? A Controlled Study Across Unseen Languages and Non-Speech Tasks

TL;DR

This study systematically evaluates neural audio codecs' ability to generalize across unseen languages and non-speech tasks, revealing that including diverse pre-training data enhances their versatility without sacrificing speech performance.

Contribution

It provides a controlled, fair comparison of NACs trained from scratch, demonstrating how pre-training data influences their generalization to new languages and non-speech applications.

Findings

NACs can generalize to unseen languages during pre-training.

Speech-only pre-trained NACs perform poorly on non-speech tasks.

Including non-speech data improves non-speech task performance while maintaining speech quality.

Abstract

This paper investigates three crucial yet underexplored aspects of the generalization capabilities of neural audio codecs (NACs): (i) whether NACs can generalize to unseen languages during pre-training, (ii) whether speech-only pre-trained NACs can effectively generalize to non-speech applications such as environmental sounds, music, and animal vocalizations, and (iii) whether incorporating non-speech data during pre-training can improve performance on both speech and non-speech tasks. Existing studies typically rely on off-the-shelf NACs for comparison, which limits insight due to variations in implementation. In this work, we train NACs from scratch using strictly controlled configurations and carefully curated pre-training data to enable fair comparisons. We conduct a comprehensive evaluation of NAC performance on both signal reconstruction quality and downstream applications using 11 metrics. Our results show that NACs can generalize to unseen languages during pre-training, speech-only pre-trained NACs exhibit degraded performance on non-speech tasks, and incorporating non-speech data during pre-training improves performance on non-speech tasks while maintaining comparable performance on speech tasks.