SPEAK WITH YOUR HANDS Using Continuous Hand Gestures to control Articulatory Speech Synthesizer

TL;DR

This paper introduces a novel hand gesture interface that translates continuous finger and wrist movements into speech sounds using articulatory synthesis, enabling speech production through hand movements alone.

Contribution

It presents a new method for controlling an articulatory speech synthesizer with hand gestures, utilizing sensor data and spline modeling to produce continuous speech without vocal tract use.

Findings

Successful translation of hand gestures into speech sounds

Smooth control of speech synthesis via spline tongue modeling

Potential for hands-only speech communication systems

Abstract

This work presents our advancements in controlling an articulatory speech synthesis engine, \textit{viz.}, Pink Trombone, with hand gestures. Our interface translates continuous finger movements and wrist flexion into continuous speech using vocal tract area-function based articulatory speech synthesis. We use Cyberglove II with 18 sensors to capture the kinematic information of the wrist and the individual fingers, in order to control a virtual tongue. The coordinates and the bending values of the sensors are then utilized to fit a spline tongue model that smoothens out the noisy values and outliers. Considering the upper palate as fixed and the spline model as the dynamically moving lower surface (tongue) of the vocal tract, we compute 1D area functional values that are fed to the Pink Trombone, generating continuous speech sounds. Therefore, by learning to manipulate one's wrist and fingers, one can learn to produce speech sounds just through one's hands, without the need for using the vocal tract.