AVE Speech: A Comprehensive Multi-Modal Dataset for Speech Recognition Integrating Audio, Visual, and Electromyographic Signals

TL;DR

The AVE Speech dataset offers a large-scale, multi-modal collection of Mandarin speech data integrating audio, video, and EMG signals, aiming to improve speech recognition especially in noisy and cross-subject scenarios.

Contribution

This paper introduces the first publicly available multi-modal Mandarin speech dataset combining audio, visual, and EMG signals for large-scale recognition tasks.

Findings

Multi-modal data significantly improves speech recognition accuracy.

Combining modalities enhances performance in noisy environments.

The dataset supports cross-subject and speaker-independent research.

Abstract

The global aging population faces considerable challenges, particularly in communication, due to the prevalence of hearing and speech impairments. To address these, we introduce the AVE speech, a comprehensive multi-modal dataset for speech recognition tasks. The dataset includes a 100-sentence Mandarin corpus with audio signals, lip-region video recordings, and six-channel electromyography (EMG) data, collected from 100 participants. Each subject read the entire corpus ten times, with each sentence averaging approximately two seconds in duration, resulting in over 55 hours of multi-modal speech data per modality. Experiments demonstrate that combining these modalities significantly improves recognition performance, particularly in cross-subject and high-noise environments. To our knowledge, this is the first publicly available sentence-level dataset integrating these three modalities for large-scale Mandarin speech recognition. We expect this dataset to drive advancements in both acoustic and non-acoustic speech recognition research, enhancing cross-modal learning and human-machine interaction.