End-to-End Lip Reading in Romanian with Cross-Lingual Domain Adaptation and Lateral Inhibition

TL;DR

This paper advances Romanian lip reading by evaluating various architectures, introducing cross-lingual domain adaptation with unlabeled data, and incorporating neural inhibition mechanisms, achieving state-of-the-art results on a limited dataset.

Contribution

It introduces a novel combination of cross-lingual domain adaptation and lateral inhibition to improve lip reading performance on underrepresented languages.

Findings

State-of-the-art results on Romanian dataset

Effective use of unlabeled English and German videos

Neural inhibition layer enhances model performance

Abstract

Lip reading or visual speech recognition has gained significant attention in recent years, particularly because of hardware development and innovations in computer vision. While considerable progress has been obtained, most models have only been tested on a few large-scale datasets. This work addresses this shortcoming by analyzing several architectures and optimizations on the underrepresented, short-scale Romanian language dataset called Wild LRRo. Most notably, we compare different backend modules, demonstrating the effectiveness of adding ample regularization methods. We obtain state-of-the-art results using our proposed method, namely cross-lingual domain adaptation and unlabeled videos from English and German datasets to help the model learn language-invariant features. Lastly, we assess the performance of adding a layer inspired by the neural inhibition mechanism.