MuCodec: Ultra Low-Bitrate Music Codec

TL;DR

MuCodec is a novel ultra low-bitrate music codec that effectively reconstructs high-fidelity music by combining acoustic and semantic features, outperforming previous methods at extremely low bitrates.

Contribution

Introduces MuCodec, a new music compression framework that leverages dual feature extraction and flow-matching for superior ultra low-bitrate reconstruction.

Findings

Achieves high-quality music reconstruction at 0.35kbps.

Outperforms existing codecs in subjective and objective metrics.

Demonstrates effective preservation of vocals and backgrounds.

Abstract

Music codecs are a vital aspect of audio codec research, and ultra low-bitrate compression holds significant importance for music transmission and generation. Due to the complexity of music backgrounds and the richness of vocals, solely relying on modeling semantic or acoustic information cannot effectively reconstruct music with both vocals and backgrounds. To address this issue, we propose MuCodec, specifically targeting music compression and reconstruction tasks at ultra low bitrates. MuCodec employs MuEncoder to extract both acoustic and semantic features, discretizes them with RVQ, and obtains Mel-VAE features via flow-matching. The music is then reconstructed using a pre-trained MEL-VAE decoder and HiFi-GAN. MuCodec can reconstruct high-fidelity music at ultra low (0.35kbps) or high bitrates (1.35kbps), achieving the best results to date in both subjective and objective metrics. Code and Demo: https://xuyaoxun.github.io/MuCodec_demo/.