Re-synchronization using the Hand Preceding Model for Multi-modal Fusion in Automatic Continuous Cued Speech Recognition

TL;DR

This paper introduces a novel re-synchronization method for multi-modal fusion in automatic continuous Cued Speech recognition, significantly improving accuracy by aligning hand and lip movements despite their asynchronous nature.

Contribution

It proposes the first re-synchronization procedure for asynchronous multi-modal fusion in CS recognition, enhancing system performance with a CNN and MSHMM framework.

Findings

Achieved a 4.6% improvement in CS phoneme recognition accuracy.

Retained 76.6% recognition correctness, surpassing previous 72.04%.

First to address asynchronous multi-modal fusion in continuous CS recognition.

Abstract

Cued Speech (CS) is an augmented lip reading complemented by hand coding, and it is very helpful to the deaf people. Automatic CS recognition can help communications between the deaf people and others. Due to the asynchronous nature of lips and hand movements, fusion of them in automatic CS recognition is a challenging problem. In this work, we propose a novel re-synchronization procedure for multi-modal fusion, which aligns the hand features with lips feature. It is realized by delaying hand position and hand shape with their optimal hand preceding time which is derived by investigating the temporal organizations of hand position and hand shape movements in CS. This re-synchronization procedure is incorporated into a practical continuous CS recognition system that combines convolutional neural network (CNN) with multi-stream hidden markov model (MSHMM). A significant improvement of about 4.6\% has been achieved retaining 76.6\% CS phoneme recognition correctness compared with the state-of-the-art architecture (72.04\%), which did not take into account the asynchrony of multi-modal fusion in CS. To our knowledge, this is the first work to tackle the asynchronous multi-modal fusion in the automatic continuous CS recognition.