Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models

TL;DR

This paper introduces Sommelier, an open-source, scalable audio pre-processing pipeline designed to improve data quality for full-duplex speech language models, addressing challenges like overlapping speech and diarization errors.

Contribution

The paper presents a novel, scalable open-source pipeline that enhances multi-turn, multi-speaker audio data processing for speech language models.

Findings

Improved handling of overlapping speech and back-channeling in data processing.

Enhanced data quality for training full-duplex speech models.

Open-source pipeline facilitates scalable data preparation.

Abstract

As the paradigm of AI shifts from text-based LLMs to Speech Language Models (SLMs), there is a growing demand for full-duplex systems capable of real-time, natural human-computer interaction. However, the development of such models is constrained by the scarcity of high-quality, multi-speaker conversational data, as existing large-scale resources are predominantly single-speaker or limited in volume. Addressing the complex dynamics of natural dialogue, such as overlapping and back-channeling remains a challenge, with standard processing pipelines suffering from diarization errors and ASR hallucinations. To bridge this gap, we present a robust and scalable open-source data processing pipeline designed for full-duplex model.