End-to-end multi-talker audio-visual ASR using an active speaker attention module

TL;DR

This paper introduces VCAM, a transformer-based end-to-end audio-visual multi-talker speech recognition model that uses visual cues to correctly assign speech to speakers, improving accuracy over previous systems.

Contribution

The paper proposes a novel visual context attention module (VCAM) that effectively resolves speaker label ambiguity in multi-talker ASR using video information.

Findings

VCAM outperforms previous audio-only and audio-visual multi-talker ASR systems.

The model is evaluated on a two-speaker overlapping speech dataset from YouTube.

Visual cues significantly improve speaker assignment accuracy.

Abstract

This paper presents a new approach for end-to-end audio-visual multi-talker speech recognition. The approach, referred to here as the visual context attention model (VCAM), is important because it uses the available video information to assign decoded text to one of multiple visible faces. This essentially resolves the label ambiguity issue associated with most multi-talker modeling approaches which can decode multiple label strings but cannot assign the label strings to the correct speakers. This is implemented as a transformer-transducer based end-to-end model and evaluated using a two speaker audio-visual overlapping speech dataset created from YouTube videos. It is shown in the paper that the VCAM model improves performance with respect to previously reported audio-only and audio-visual multi-talker ASR systems.