SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model

TL;DR

This paper introduces SALM-Duplex, a novel speech-to-speech model that enables real-time, continuous dialogue with improved reasoning and turn-taking, achieved with less data and no speech pretraining.

Contribution

It presents the first duplex S2S model with continuous input, channel fusion, and reduced bitrate, simplifying development and enhancing performance over previous models.

Findings

Outperforms previous duplex models in reasoning and turn-taking.

Halves bitrate to 0.6 kbps compared to prior work.

Requires less speech data by skipping speech pretraining.

Abstract

Spoken dialogue is an intuitive form of human-computer interaction, yet current speech language models often remain constrained to turn-based exchanges, lacking real-time adaptability such as user barge-in. We propose a novel duplex speech to speech (S2S) architecture featuring continuous user inputs and codec agent outputs with channel fusion that directly models simultaneous user and agent streams. Using a pretrained streaming encoder for user input enables the first duplex S2S model without requiring speech pretrain. Separate architectures for agent and user modeling facilitate codec fine-tuning for better agent voices and halve the bitrate (0.6 kbps) compared to previous works. Experimental results show that the proposed model outperforms previous duplex models in reasoning, turn-taking, and barge-in abilities. The model requires significantly less speech data, as speech pretrain is skipped, which markedly simplifies the process of building a duplex S2S model from any LLMs. Finally, it is the first openly available duplex S2S model with training and inference code to foster reproducibility.