End-to-end Silent Speech Recognition with Acoustic Sensing

TL;DR

This paper introduces a non-invasive silent speech recognition system using acoustic reflections captured by smartphones, employing deep learning for accurate speech recognition without audible sound.

Contribution

It proposes an end-to-end acoustic sensing framework combining CNN and attention-based networks for silent speech recognition, a novel approach in the field.

Findings

Achieved 8.4% word error rate in speaker-independent settings

Demonstrated robustness with 8.1% error on unseen sentences

Utilized smartphone hardware for non-invasive silent speech capture

Abstract

Silent speech interfaces (SSI) has been an exciting area of recent interest. In this paper, we present a non-invasive silent speech interface that uses inaudible acoustic signals to capture people's lip movements when they speak. We exploit the speaker and microphone of the smartphone to emit signals and listen to their reflections, respectively. The extracted phase features of these reflections are fed into the deep learning networks to recognize speech. And we also propose an end-to-end recognition framework, which combines the CNN and attention-based encoder-decoder network. Evaluation results on a limited vocabulary (54 sentences) yield word error rates of 8.4% in speaker-independent and environment-independent settings, and 8.1% for unseen sentence testing.