Personalized One-Shot Lipreading for an ALS Patient

TL;DR

This paper introduces a personalized one-shot lipreading method for ALS patients that uses synthetic data and domain adaptation to achieve high accuracy with minimal real data, aiding communication for medical patients.

Contribution

The work presents a novel personalized lipreading approach for ALS patients using one-shot learning and synthetic data augmentation with domain adaptation techniques.

Findings

Achieved 83.2% top-5 accuracy on ALS patient data.

Outperformed comparable methods with 62.6% accuracy.

Extended approach to people with hearing impairment.

Abstract

Lipreading or visually recognizing speech from the mouth movements of a speaker is a challenging and mentally taxing task. Unfortunately, multiple medical conditions force people to depend on this skill in their day-to-day lives for essential communication. Patients suffering from Amyotrophic Lateral Sclerosis (ALS) often lose muscle control, consequently their ability to generate speech and communicate via lip movements. Existing large datasets do not focus on medical patients or curate personalized vocabulary relevant to an individual. Collecting a large-scale dataset of a patient, needed to train mod-ern data-hungry deep learning models is, however, extremely challenging. In this work, we propose a personalized network to lipread an ALS patient using only one-shot examples. We depend on synthetically generated lip movements to augment the one-shot scenario. A Variational Encoder based domain adaptation technique is used to bridge the real-synthetic domain gap. Our approach significantly improves and achieves high top-5accuracy with 83.2% accuracy compared to 62.6% achieved by comparable methods for the patient. Apart from evaluating our approach on the ALS patient, we also extend it to people with hearing impairment relying extensively on lip movements to communicate.