CSLNSpeech: solving extended speech separation problem with the help of Chinese sign language

TL;DR

This paper introduces CSLNSpeech, a deep learning framework that leverages audio, face, and sign language modalities to improve speech separation, especially aiding hearing-impaired individuals, and demonstrates its effectiveness on a new large-scale dataset and existing benchmarks.

Contribution

The paper presents a novel multi-modal speech separation model incorporating sign language, along with a large-scale Chinese Sign Language News Speech dataset for training and evaluation.

Findings

The model outperforms traditional audio-visual systems in accuracy and robustness.

Sign language alone can effectively supervise speech separation.

The framework achieves competitive results on multiple datasets.

Abstract

Previous audio-visual speech separation methods use the synchronization of the speaker's facial movement and speech in the video to supervise the speech separation in a self-supervised way. In this paper, we propose a model to solve the speech separation problem assisted by both face and sign language, which we call the extended speech separation problem. We design a general deep learning network for learning the combination of three modalities, audio, face, and sign language information, for better solving the speech separation problem. To train the model, we introduce a large-scale dataset named the Chinese Sign Language News Speech (CSLNSpeech) dataset, in which three modalities of audio, face, and sign language coexist. Experiment results show that the proposed model has better performance and robustness than the usual audio-visual system. Besides, sign language modality can also be used alone to supervise speech separation tasks, and the introduction of sign language is helpful for hearing-impaired people to learn and communicate. Last, our model is a general speech separation framework and can achieve very competitive separation performance on two open-source audio-visual datasets. The code is available at https://github.com/iveveive/SLNSpeech