Scaling Transformers for Low-Bitrate High-Quality Speech Coding

TL;DR

This paper demonstrates that scaling transformer architectures with a Finite Scalar Quantization bottleneck can achieve state-of-the-art low-bitrate speech coding quality, surpassing existing methods in objective and subjective evaluations.

Contribution

It introduces a large-scale transformer model with FSQ bottleneck for low-bitrate speech coding, achieving unprecedented speech quality at extremely low bit-rates.

Findings

Outperforms existing baselines in objective tests

Achieves state-of-the-art speech quality at 400-700 bits/sec

Demonstrates effectiveness of scaled transformers in speech tokenization

Abstract

The tokenization of speech with neural audio codec models is a vital part of modern AI pipelines for the generation or understanding of speech, alone or in a multimodal context. Traditionally such tokenization models have concentrated on low parameter-count architectures using only components with strong inductive biases. In this work we show that by scaling a transformer architecture with large parameter count to this problem, and applying a flexible Finite Scalar Quantization (FSQ) based bottleneck, it is possible to reach state-of-the-art speech quality at extremely low bit-rates of $400$ or $700$ bits-per-second. The trained models strongly out-perform existing baselines in both objective and subjective tests.