OmniCodec: Low Frame Rate Universal Audio Codec with Semantic-Acoustic Disentanglement

TL;DR

OmniCodec is a universal low frame rate neural audio codec that effectively disentangles semantic and acoustic information, achieving superior reconstruction quality and more meaningful representations across diverse audio domains.

Contribution

It introduces a hierarchical multi-codebook design with semantic-acoustic decoupling and a self-guidance strategy, enabling low frame rate modeling for various audio types.

Findings

Outperforms Mimi codec at the same bitrate

Provides more semantically informative representations

Enhances downstream audio generation tasks

Abstract

Large Language Models (LLMs) have advanced audio generation through discrete representation learning. However, most existing neural codecs focus on speech and emphasize reconstruction fidelity, overlooking unified low frame rate modeling across diverse audio domains, including speech, music, and general sound. Moreover, high reconstruction quality does not necessarily yield semantically informative representations, limiting effectiveness in downstream generation tasks. We propose OmniCodec, a universal neural audio codec tailored for low frame rate. It adopts a hierarchical multi-codebook design with semantic-acoustic decoupling by leveraging the audio encoder of the pre-trained understanding model, along with a self-guidance strategy to improve codebook utilization and reconstruction. Compared with the Mimi codec, experiments show that OmniCodec achieves outstanding performance at the same bitrate, delivering superior reconstruction quality while also providing more semantically informative representations that benefit downstream generation tasks. Our model and code will be open-sourced. Our demo page is available.