NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference

TL;DR

NanoCodec is a novel low frame-rate audio codec that enables high-quality, ultra-fast speech LLM inference by significantly reducing autoregressive steps, outperforming existing codecs in quality and efficiency.

Contribution

We introduce NanoCodec, a new audio codec operating at 12.5 FPS that improves speech compression quality and efficiency for LLM applications, setting a new benchmark.

Findings

NanoCodec achieves high-quality compression at 12.5 FPS.

NanoCodec outperforms related codecs across various bitrates.

NanoCodec enables faster and more efficient speech LLM inference.

Abstract

Large Language Models (LLMs) have significantly advanced audio processing by leveraging audio codecs to discretize audio into tokens, enabling the application of language modeling techniques to speech data. However, existing audio codecs often operate at high frame rates, leading to slow training and inference, particularly for autoregressive models. To address this, there is growing interest in low frame-rate audio codecs, which reduce the number of autoregressive steps required to generate one second of audio. In this paper, we conduct ablation studies to examine the impact of frame rate, bitrate, and causality on codec reconstruction quality. Based on our findings, we introduce NanoCodec, a state-of-the-art audio codec that achieves high-quality compression at just 12.5 frames per second (FPS). NanoCodec outperforms related works across various bitrate ranges, establishing a new benchmark for low-latency and efficient Speech LLM training and inference.