Lip-Listening: Mixing Senses to Understand Lips using Cross Modality Knowledge Distillation for Word-Based Models

TL;DR

This paper introduces a novel approach to lipreading by transferring audio speech recognition knowledge to visual models using cross-modality knowledge distillation, significantly improving lipreading accuracy.

Contribution

It develops a new framework combining sequence and frame-level knowledge distillation, leveraging audio data during training of visual speech recognition models, and proposes an efficient Gaussian averaging technique.

Findings

Achieved 88.64% accuracy on LRW dataset.

Set a new benchmark for lipreading performance.

Demonstrated the effectiveness of cross-modality knowledge transfer.

Abstract

In this work, we propose a technique to transfer speech recognition capabilities from audio speech recognition systems to visual speech recognizers, where our goal is to utilize audio data during lipreading model training. Impressive progress in the domain of speech recognition has been exhibited by audio and audio-visual systems. Nevertheless, there is still much to be explored with regards to visual speech recognition systems due to the visual ambiguity of some phonemes. To this end, the development of visual speech recognition models is crucial given the instability of audio models. The main contributions of this work are i) building on recent state-of-the-art word-based lipreading models by integrating sequence-level and frame-level Knowledge Distillation (KD) to their systems; ii) leveraging audio data during training visual models, a feat which has not been utilized in prior word-based work; iii) proposing the Gaussian-shaped averaging in frame-level KD, as an efficient technique that aids the model in distilling knowledge at the sequence model encoder. This work proposes a novel and competitive architecture for lip-reading, as we demonstrate a noticeable improvement in performance, setting a new benchmark equals to 88.64% on the LRW dataset.