UniSE: A Unified Framework for Decoder-only Autoregressive LM-based Speech Enhancement

TL;DR

UniSE introduces a unified decoder-only autoregressive language model framework that effectively handles various speech enhancement tasks, demonstrating competitive performance across multiple benchmarks.

Contribution

This work is the first to verify the effectiveness of autoregressive language models in unifying different speech enhancement sub-tasks.

Findings

Achieves competitive results on several benchmarks.

Demonstrates the capacity of LMs to unify multiple SE tasks.

Shows compatibility of distinct learning patterns in a single framework.

Abstract

The development of neural audio codecs (NACs) has largely promoted applications of language models (LMs) to speech processing and understanding. However, there lacks the verification on the effectiveness of autoregressive (AR) LMbased models in unifying different sub-tasks of speech enhancement (SE). In this work, we propose UniSE, a unified decoder-only LM-based framework to handle different SE tasks including speech restoration, target speaker extraction and speech separation. It takes input speech features as conditions and generates discrete tokens of the target speech using AR modeling, which facilitates a compatibility between distinct learning patterns of multiple tasks. Experiments on several benchmarks indicate the proposed UniSE can achieve competitive performance compared to discriminative and generative baselines, showing the capacity of LMs in unifying SE tasks. The demo page is available here: https://github.com/hyyan2k/UniSE.