Towards Pose-invariant Lip-Reading

TL;DR

This paper introduces a pose-invariant lip-reading framework using synthetic data generated by a 3D Morphable Model, significantly improving recognition accuracy across various mouth poses.

Contribution

The novel use of 3DMM to generate synthetic multi-pose facial data for training lip-reading models, enhancing performance in non-frontal views.

Findings

Outperforms previous methods on non-frontal views

Achieves up to 20.64% improvement in extreme poses

Improves cross-database word recognition accuracy by 2.55%

Abstract

Lip-reading models have been significantly improved recently thanks to powerful deep learning architectures. However, most works focused on frontal or near frontal views of the mouth. As a consequence, lip-reading performance seriously deteriorates in non-frontal mouth views. In this work, we present a framework for training pose-invariant lip-reading models on synthetic data instead of collecting and annotating non-frontal data which is costly and tedious. The proposed model significantly outperforms previous approaches on non-frontal views while retaining the superior performance on frontal and near frontal mouth views. Specifically, we propose to use a 3D Morphable Model (3DMM) to augment LRW, an existing large-scale but mostly frontal dataset, by generating synthetic facial data in arbitrary poses. The newly derived dataset, is used to train a state-of-the-art neural network for lip-reading. We conducted a cross-database experiment for isolated word recognition on the LRS2 dataset, and reported an absolute improvement of 2.55%. The benefit of the proposed approach becomes clearer in extreme poses where an absolute improvement of up to 20.64% over the baseline is achieved.