SLM-SS: Speech Language Model for Generative Speech Separation

TL;DR

SLM-SS introduces a novel speech language model-based approach to speech separation, significantly improving intelligibility and downstream task performance by framing the problem as sequence generation with autoregressive and non-autoregressive models.

Contribution

The paper presents a new method applying speech language models to speech separation, enhancing intelligibility and efficiency over existing neural network-based approaches.

Findings

Better preservation of speech intelligibility.

Improved performance in downstream speech recognition tasks.

Effective sequence generation with combined autoregressive and non-autoregressive models.

Abstract

Speech separation (SS) has advanced significantly with neural network-based methods, showing improved performance on signal-level metrics. However, these methods often struggle to maintain speech intelligibility in the separated signals, which can negatively affect the performance of downstream tasks such as speech recognition. In this work, we propose SLM-SS, a novel approach that applies speech language models to SS, aiming to enhance the intelligibility and coherence of the separated signals. We frame SS as discrete multi-codebook sequence generation, using Encoder-Decoder models to map quantized speech mixtures to target tokens. In addition to the autoregressive modeling strategy, we introduce a non-autoregressive model to improve decoding efficiency for residual tokens. Experimental results on the LibriMix dataset demonstrate that our approach shows significantly better preservation of speech intelligibility, leading to improved linguistic consistency in a variety of downstream tasks compared to existing approaches.