LMCodec: A Low Bitrate Speech Codec With Causal Transformer Models

TL;DR

LMCodec is a novel low bitrate speech codec utilizing causal Transformer models to efficiently encode and predict speech tokens, achieving high audio quality comparable to higher bitrate codecs through hierarchical quantization and entropy coding.

Contribution

It introduces a causal neural speech codec combining hierarchical vector quantization with Transformer-based prediction and entropy coding, enabling high-quality low bitrate speech transmission.

Findings

Subjective tests show comparable quality to higher bitrate codecs.

Hierarchical token encoding improves compression efficiency.

Transformer-based prediction reduces transmitted codes.

Abstract

We introduce LMCodec, a causal neural speech codec that provides high quality audio at very low bitrates. The backbone of the system is a causal convolutional codec that encodes audio into a hierarchy of coarse-to-fine tokens using residual vector quantization. LMCodec trains a Transformer language model to predict the fine tokens from the coarse ones in a generative fashion, allowing for the transmission of fewer codes. A second Transformer predicts the uncertainty of the next codes given the past transmitted codes, and is used to perform conditional entropy coding. A MUSHRA subjective test was conducted and shows that the quality is comparable to reference codecs at higher bitrates. Example audio is available at https://mjenrungrot.github.io/chrome-media-audio-papers/publications/lmcodec.