Phoneme-Level Visual Speech Recognition via Point-Visual Fusion and Language Model Reconstruction

TL;DR

This paper introduces a phoneme-level visual speech recognition framework that combines visual and landmark features with language model reconstruction, significantly improving accuracy in lip-reading tasks.

Contribution

It proposes a novel two-stage phoneme-based framework with visual-landmark fusion and language model reconstruction, reducing errors caused by viseme ambiguity.

Findings

Achieves 17.4% WER on LRS2 dataset

Achieves 21.0% WER on LRS3 dataset

Outperforms existing visual speech recognition methods

Abstract

Visual Automatic Speech Recognition (V-ASR) is a challenging task that involves interpreting spoken language solely from visual information, such as lip movements and facial expressions. This task is notably challenging due to the absence of auditory cues and the visual ambiguity of phonemes that exhibit similar visemes-distinct sounds that appear identical in lip motions. Existing methods often aim to predict words or characters directly from visual cues, but they commonly suffer from high error rates due to viseme ambiguity and require large amounts of pre-training data. We propose a novel phoneme-based two-stage framework that fuses visual and landmark motion features, followed by an LLM model for word reconstruction to address these challenges. Stage 1 consists of V-ASR, which outputs the predicted phonemes, thereby reducing training complexity. Meanwhile, the facial landmark features address speaker-specific facial characteristics. Stage 2 comprises an encoder-decoder LLM model, NLLB, that reconstructs the output phonemes back to words. Besides using a large visual dataset for deep learning fine-tuning, our PV-ASR method demonstrates superior performance by achieving 17.4% WER on the LRS2 and 21.0% WER on the LRS3 dataset.