Some observations on computer lip-reading: moving from the dream to the reality

TL;DR

This paper critically examines common assumptions in computer lip-reading, revealing that factors like video quality are less limiting than previously thought, but questioning the effectiveness of visemes as recognition units.

Contribution

It challenges the prevailing reliance on visemes for lip-reading and suggests that alternative units may improve system performance.

Findings

Video resolution and lighting have limited impact on lip-reading accuracy.

Visemes may not be the optimal units for recognition in modern systems.

Practical factors are less constraining than theoretical assumptions.

Abstract

In the quest for greater computer lip-reading performance there are a number of tacit assumptions which are either present in the datasets (high resolution for example) or in the methods (recognition of spoken visual units called visemes for example). Here we review these and other assumptions and show the surprising result that computer lip-reading is not heavily constrained by video resolution, pose, lighting and other practical factors. However, the working assumption that visemes, which are the visual equivalent of phonemes, are the best unit for recognition does need further examination. We conclude that visemes, which were defined over a century ago, are unlikely to be optimal for a modern computer lip-reading system.