Landmark Guided Visual Feature Extractor for Visual Speech Recognition with Limited Resource

TL;DR

This paper introduces a landmark guided visual feature extractor for visual speech recognition that leverages facial landmarks and a multi-graph convolutional network to improve accuracy with limited data and reduce user-specific feature influence.

Contribution

It proposes a novel landmark guided visual feature extractor using a multi-graph convolutional network and multi-level lip dynamic fusion to enhance recognition performance with limited data.

Findings

Performs well with limited training data.

Improves accuracy on unseen speakers.

Reduces influence of visual disturbances.

Abstract

Visual speech recognition is a technique to identify spoken content in silent speech videos, which has raised significant attention in recent years. Advancements in data-driven deep learning methods have significantly improved both the speed and accuracy of recognition. However, these deep learning methods can be effected by visual disturbances, such as lightning conditions, skin texture and other user-specific features. Data-driven approaches could reduce the performance degradation caused by these visual disturbances using models pretrained on large-scale datasets. But these methods often require large amounts of training data and computational resources, making them costly. To reduce the influence of user-specific features and enhance performance with limited data, this paper proposed a landmark guided visual feature extractor. Facial landmarks are used as auxiliary information to aid in training the visual feature extractor. A spatio-temporal multi-graph convolutional network is designed to fully exploit the spatial locations and spatio-temporal features of facial landmarks. Additionally, a multi-level lip dynamic fusion framework is introduced to combine the spatio-temporal features of the landmarks with the visual features extracted from the raw video frames. Experimental results show that this approach performs well with limited data and also improves the model's accuracy on unseen speakers.