Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspective

TL;DR

This paper investigates why speech language models struggle with semantic coherence, analyzing the effects of phonetic information, sequence length, and paralinguistic features through modality evolution.

Contribution

It systematically examines the influence of three key factors on speech language model performance, providing insights for improving end-to-end SLM development.

Findings

Paralinguistic features have the most significant impact on lexical modeling.

Sequence length affects syntactic and semantic modeling more than phonetic information.

Phonetic information has a relatively minor impact on semantic coherence.

Abstract

Although text-based large language models exhibit human-level writing ability and remarkable intelligence, speech language models (SLMs) still struggle to generate semantically coherent outputs. There are several potential reasons for this performance degradation: (A) speech tokens mainly provide phonetic information rather than semantic information, (B) the length of speech sequences is much longer than that of text sequences, and (C) paralinguistic information, such as prosody, introduces additional complexity and variability. In this paper, we explore the influence of three key factors separately by transiting the modality from text to speech in an evolving manner. Our findings reveal that the impact of the three factors varies. Factor A has a relatively minor impact, factor B influences syntactical and semantic modeling more obviously, and factor C exerts the most significant impact, particularly in the basic lexical modeling. Based on these findings, we provide insights into the unique challenges of training SLMs and highlight pathways to develop more effective end-to-end SLMs.