Can LLMs Understand Unvoiced Speech? Exploring EMG-to-Text Conversion with LLMs

TL;DR

This paper investigates using large language models to interpret unvoiced electromyography signals for speech, introducing a novel EMG adaptor that enables LLMs to understand unvoiced speech with minimal data.

Contribution

It proposes a new EMG adaptor module that maps EMG features into LLM input space, enabling unvoiced speech recognition without paired voiced data.

Findings

Achieved an average WER of 0.49 on unvoiced EMG-to-text task.

Improved performance by nearly 20% over specialized models with only six minutes of data.

Demonstrated LLMs can be extended to understand biosignals like unvoiced EMG.

Abstract

Unvoiced electromyography (EMG) is an effective communication tool for individuals unable to produce vocal speech. However, most prior methods rely on paired voiced and unvoiced EMG signals, along with speech data, for EMG-to-text conversion, which is not practical for such individuals. Given the rise of large language models (LLMs) in speech recognition, we explore their potential to understand unvoiced speech. To this end, we address the challenge of learning from unvoiced EMG alone and propose a novel EMG adaptor module that maps EMG features into an LLM's input space, achieving an average word error rate (WER) of 0.49 on a closed-vocabulary unvoiced EMG-to-text task. Even with a conservative data availability of just six minutes, our approach improves performance over specialized models by nearly 20%. While LLMs have been shown to be extendable to new language modalities -- such as audio -- understanding articulatory biosignals like unvoiced EMG remains more challenging. This work takes a crucial first step toward enabling LLMs to comprehend unvoiced speech using surface EMG.