Visual Features for Context-Aware Speech Recognition

TL;DR

This paper explores how integrating visual context, such as objects and scenes detected in videos, can improve the accuracy of speech recognition systems by reducing word error rates.

Contribution

It introduces a method to adapt both acoustic and language models in speech recognition using visual context from video content, enhancing transcription accuracy.

Findings

Significant reduction in word error rate when using visual context

Improved perplexity in language models with scene/object information

Applicable to various speech processing domains like robotics and multimedia indexing

Abstract

Automatic transcriptions of consumer-generated multi-media content such as "Youtube" videos still exhibit high word error rates. Such data typically occupies a very broad domain, has been recorded in challenging conditions, with cheap hardware and a focus on the visual modality, and may have been post-processed or edited. In this paper, we extend our earlier work on adapting the acoustic model of a DNN-based speech recognition system to an RNN language model and show how both can be adapted to the objects and scenes that can be automatically detected in the video. We are working on a corpus of "how-to" videos from the web, and the idea is that an object that can be seen ("car"), or a scene that is being detected ("kitchen") can be used to condition both models on the "context" of the recording, thereby reducing perplexity and improving transcription. We achieve good improvements in both cases and compare and analyze the respective reductions in word error rate. We expect that our results can be used for any type of speech processing in which "context" information is available, for example in robotics, man-machine interaction, or when indexing large audio-visual archives, and should ultimately help to bring together the "video-to-text" and "speech-to-text" communities.