Code Drift: Towards Idempotent Neural Audio Codecs

TL;DR

This paper investigates the property of idempotence in neural audio codecs, identifies issues with stability under multiple encodings, and proposes a fine-tuning method to improve idempotence without harming performance.

Contribution

It introduces a method to enhance idempotence in neural audio codecs through fine-tuning, addressing a previously overlooked property.

Findings

Neural codecs show varied idempotence levels.

Fine-tuning can significantly improve idempotence.

Improved idempotence does not harm downstream generative tasks.

Abstract

Neural codecs have demonstrated strong performance in high-fidelity compression of audio signals at low bitrates. The token-based representations produced by these codecs have proven particularly useful for generative modeling. While much research has focused on improvements in compression ratio and perceptual transparency, recent works have largely overlooked another desirable codec property -- idempotence, the stability of compressed outputs under multiple rounds of encoding. We find that state-of-the-art neural codecs exhibit varied degrees of idempotence, with some degrading audio outputs significantly after as few as three encodings. We investigate possible causes of low idempotence and devise a method for improving idempotence through fine-tuning a codec model. We then examine the effect of idempotence on a simple conditional generative modeling task, and find that increased idempotence can be achieved without negatively impacting downstream modeling performance -- potentially extending the usefulness of neural codecs for practical file compression and iterative generative modeling workflows.