OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model

TL;DR

OpenS2S is a fully open-source, end-to-end empathetic speech language model designed for transparent research and low-latency, emotionally expressive speech interactions, utilizing automated data synthesis and scalable training methods.

Contribution

It introduces a novel open-source empathetic speech model with an automated data pipeline and streaming decoding architecture for low-latency, expressive speech generation.

Findings

Achieved low-latency speech generation with streaming interleaved decoding.

Constructed a diverse, high-quality empathetic speech dataset with minimal supervision.

Released comprehensive open-source resources including model, dataset, and code.

Abstract

Empathetic interaction is a cornerstone of human-machine communication, due to the need for understanding speech enriched with paralinguistic cues and generating emotional and expressive responses. However, the most powerful empathetic LSLMs are increasingly closed off, leaving the crucial details about the architecture, data and development opaque to researchers. Given the critical need for transparent research into the LSLMs and empathetic behavior, we present OpenS2S, a fully open-source, transparent and end-to-end LSLM designed to enable empathetic speech interactions. Based on our empathetic speech-to-text model BLSP-Emo, OpenS2S further employs a streaming interleaved decoding architecture to achieve low-latency speech generation. To facilitate end-to-end training, OpenS2S incorporates an automated data construction pipeline that synthesizes diverse, high-quality empathetic speech dialogues at low cost. By leveraging large language models to generate empathetic content and controllable text-to-speech systems to introduce speaker and emotional variation, we construct a scalable training corpus with rich paralinguistic diversity and minimal human supervision. We release the fully open-source OpenS2S model, including the dataset, model weights, pre-training and fine-tuning codes, to empower the broader research community and accelerate innovation in empathetic speech systems. The project webpage can be accessed at https://casia-lm.github.io/OpenS2S