Optimising Neural Speech Codecs for 300bps Communication using Reinforcement Learning

TL;DR

This paper introduces ClariCodec, a neural speech codec optimized for ultra-low bitrate communication at 300 bps, using reinforcement learning to improve intelligibility measured by word error rate.

Contribution

It presents a novel RL-based fine-tuning method for neural speech codecs that significantly enhances intelligibility at extremely low bitrates without sacrificing perceptual quality.

Findings

ClariCodec achieves 4.64% WER at 300 bps without RL.

RL fine-tuning reduces WER to 3.55% on test-clean.

The method improves intelligibility by 23% relative WER reduction.

Abstract

In bandwidth-constrained communication such as satellite and underwater channels, speech must often be transmitted at ultra-low bitrates where intelligibility is the primary objective. At such extreme compression levels, codecs trained with acoustic reconstruction losses tend to allocate bits to perceptual detail, leading to substantial degradation in word error rate (WER). This paper proposes ClariCodec, a neural speech codec operating at 300 bit per second (bps) that reformulates quantisation as a stochastic policy, enabling reinforcement learning (RL)-based optimisation of intelligibility. Specifically, the encoder is fine-tuned using WER-driven rewards while the acoustic reconstruction pipeline remains frozen. Even without RL, ClariCodec achieves 4.64% WER on the LibriSpeech test-clean set at 300 bps, already competitive with codecs operating at higher bitrates. Further RL fine-tuning reduces WER to 3.55% on test-clean and 10.4% on test-other, corresponding to a 23% relative reduction while preserving perceptual quality.