Taming Audio VAEs via Target-KL Regularization

TL;DR

This paper introduces a target-KL regularization framework for audio VAEs, enabling controlled compression and improved rate-distortion trade-offs in audio generation tasks.

Contribution

It presents a novel method for training audio VAEs at specific bitrates, facilitating direct comparison with neural audio codecs and optimizing generation quality.

Findings

Target-KL regularization helps identify optimal compression rates.

The framework enables construction of rate-distortion curves for audio VAEs.

Sweeping compression rates improves text-to-sound generation quality.

Abstract

Latent diffusion models have emerged as the dominant paradigm for many generation tasks including audio generation such as text-to-audio, text-to-music and text-to-speech. A key component of latent diffusion is an autoencoder (VAE) that compresses high-dimensional signals into a low frame rate continuous representation that is conducive for downstream prediction. Regularizing these VAEs is challenging, as there is a trade-off between over-regularized (poor output quality) and under-regularized (difficult to predict) latent representations. We propose a framework for studying this trade-off through compression and train Audio VAEs at specific bitrates via target-KL regularization. This allows direct comparison to well-studied discrete neural audio codec models, and the construction of rate-distortion curves for audio VAEs. We evaluate the impact of target-KL regularization on text-to-sound generation and find that sweeping compression rates is helpful in identifying the optimal generation setting.