GOAT-SLM: A Spoken Language Model with Paralinguistic and Speaker Characteristic Awareness

TL;DR

GOAT-SLM is a novel spoken language model that incorporates paralinguistic and speaker characteristics, enabling more natural and socially aware speech interactions by modeling beyond just linguistic content.

Contribution

It introduces a dual-modality architecture and a staged training strategy to effectively integrate linguistic, paralinguistic, and speaker information in spoken language modeling.

Findings

Outperforms existing models in emotion, dialect, and age-sensitive tasks.

Achieves balanced performance across semantic and non-semantic tasks.

Demonstrates robustness in handling diverse speaker characteristics.

Abstract

Recent advances in end-to-end spoken language models (SLMs) have significantly improved the ability of AI systems to engage in natural spoken interactions. However, most existing models treat speech merely as a vehicle for linguistic content, often overlooking the rich paralinguistic and speaker characteristic cues embedded in human speech, such as dialect, age, emotion, and non-speech vocalizations. In this work, we introduce GOAT-SLM, a novel spoken language model with paralinguistic and speaker characteristic awareness, designed to extend spoken language modeling beyond text semantics. GOAT-SLM adopts a dual-modality head architecture that decouples linguistic modeling from acoustic realization, enabling robust language understanding while supporting expressive and adaptive speech generation. To enhance model efficiency and versatility, we propose a modular, staged training strategy that progressively aligns linguistic, paralinguistic, and speaker characteristic information using large-scale speech-text corpora. Experimental results on TELEVAL, a multi-dimensional evaluation benchmark, demonstrate that GOAT-SLM achieves well-balanced performance across both semantic and non-semantic tasks, and outperforms existing open-source models in handling emotion, dialectal variation, and age-sensitive interactions. This work highlights the importance of modeling beyond linguistic content and advances the development of more natural, adaptive, and socially aware spoken language systems.