ASR is all you need: cross-modal distillation for lip reading

TL;DR

This paper introduces a novel cross-modal distillation approach that enables training effective lip reading models without human transcriptions by leveraging large-scale audio speech recognition models and unlabelled video data.

Contribution

It demonstrates that ground truth transcriptions are unnecessary, leverages unlabelled video data, accelerates training, and achieves state-of-the-art lip reading results.

Findings

Ground truth transcriptions are not needed for training.

Unlabelled video data improves lip reading performance.

Distillation speeds up training significantly.

Abstract

The goal of this work is to train strong models for visual speech recognition without requiring human annotated ground truth data. We achieve this by distilling from an Automatic Speech Recognition (ASR) model that has been trained on a large-scale audio-only corpus. We use a cross-modal distillation method that combines Connectionist Temporal Classification (CTC) with a frame-wise cross-entropy loss. Our contributions are fourfold: (i) we show that ground truth transcriptions are not necessary to train a lip reading system; (ii) we show how arbitrary amounts of unlabelled video data can be leveraged to improve performance; (iii) we demonstrate that distillation significantly speeds up training; and, (iv) we obtain state-of-the-art results on the challenging LRS2 and LRS3 datasets for training only on publicly available data.