ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models

TL;DR

ProsodyLM introduces a novel tokenization scheme that enables speech language models to better learn and process prosody information, leading to emerging capabilities in understanding and generating nuanced speech features.

Contribution

The paper proposes a new prosody-aware tokenization method that enhances speech language models' ability to learn prosody during pre-training.

Findings

ProsodyLM captures prosody nuances like contrastive focus.

It understands emotion and stress in speech.

It maintains prosody consistency over long contexts.

Abstract

Speech language models refer to language models with speech processing and understanding capabilities. One key desirable capability for speech language models is the ability to capture the intricate interdependency between content and prosody. The existing mainstream paradigm of training speech language models, which converts speech into discrete tokens before feeding them into LLMs, is sub-optimal in learning prosody information -- we find that the resulting LLMs do not exhibit obvious emerging prosody processing capabilities via pre-training alone. To overcome this, we propose ProsodyLM, which introduces a simple tokenization scheme amenable to learning prosody. Each speech utterance is first transcribed into text, followed by a sequence of word-level prosody tokens. Compared with conventional speech tokenization schemes, the proposed tokenization scheme retains more complete prosody information, and is more understandable to text-based LLMs. We find that ProsodyLM can learn surprisingly diverse emerging prosody processing capabilities through pre-training alone, ranging from harnessing the prosody nuances in generated speech, such as contrastive focus, understanding emotion and stress in an utterance, to maintaining prosody consistency in long contexts.