A Variational Framework for Improving Naturalness in Generative Spoken Language Models

TL;DR

This paper introduces a variational framework that automatically encodes prosodic and paralinguistic speech attributes to improve the naturalness of generative spoken language models, reducing manual feature engineering.

Contribution

It presents an end-to-end variational method that learns to encode continuous speech attributes, enhancing naturalness without manual feature selection.

Findings

Generated speech is rated more natural by human evaluators.

The method reduces reliance on hand-engineered features.

Code and models are publicly available.

Abstract

The success of large language models in text processing has inspired their adaptation to speech modeling. However, since speech is continuous and complex, it is often discretized for autoregressive modeling. Speech tokens derived from self-supervised models (known as semantic tokens) typically focus on the linguistic aspects of speech but neglect prosodic information. As a result, models trained on these tokens can generate speech with reduced naturalness. Existing approaches try to fix this by adding pitch features to the semantic tokens. However, pitch alone cannot fully represent the range of paralinguistic attributes, and selecting the right features requires careful hand-engineering. To overcome this, we propose an end-to-end variational approach that automatically learns to encode these continuous speech attributes to enhance the semantic tokens. Our approach eliminates the need for manual extraction and selection of paralinguistic features. Moreover, it produces preferred speech continuations according to human raters. Code, samples and models are available at https://github.com/b04901014/vae-gslm.