Interactive decoding of words from visual speech recognition models

TL;DR

This paper introduces an interactive decoding approach for visual speech recognition that incorporates user input at each word position to improve accuracy, demonstrated through automated evaluation with simulated user interactions.

Contribution

It presents a novel interactive decoding framework that allows user input during visual speech recognition, enhancing performance over traditional phoneme-to-word pipelines.

Findings

Interactive decoding improves recognition accuracy

Simulated user input effectively guides decoding process

Method shows promise for real-time text input applications

Abstract

This work describes an interactive decoding method to improve the performance of visual speech recognition systems using user input to compensate for the inherent ambiguity of the task. Unlike most phoneme-to-word decoding pipelines, which produce phonemes and feed these through a finite state transducer, our method instead expands words in lockstep, facilitating the insertion of interaction points at each word position. Interaction points enable us to solicit input during decoding, allowing users to interactively direct the decoding process. We simulate the behavior of user input using an oracle to give an automated evaluation, and show promise for the use of this method for text input.