Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment

TL;DR

This paper introduces MUTUD, a framework that enables effective audiovisual speech processing by training with multiple modalities but deploying with fewer, reducing computational costs and maintaining high performance.

Contribution

The paper presents a novel MUTUD framework with TAME module that estimates missing modality information, improving unimodal deployment performance while leveraging multimodal training.

Findings

MUTUD reduces the performance gap between multimodal and unimodal models.

It achieves up to 80% reduction in model size and compute.

Effective across various audiovisual speech tasks.

Abstract

Building reliable speech systems often requires combining multiple modalities, like audio and visual cues. While such multimodal solutions frequently lead to improvements in performance and may even be critical in certain cases, they come with several constraints such as increased sensory requirements, computational cost, and modality synchronization, to mention a few. These challenges constrain the direct uses of these multimodal solutions in real-world applications. In this work, we develop approaches where the learning happens with all available modalities but the deployment or inference is done with just one or reduced modalities. To do so, we propose a Multimodal Training and Unimodal Deployment (MUTUD) framework which includes a Temporally Aligned Modality feature Estimation (TAME) module that can estimate information from missing modality using modalities present during inference. This innovative approach facilitates the integration of information across different modalities, enhancing the overall inference process by leveraging the strengths of each modality to compensate for the absence of certain modalities during inference. We apply MUTUD to various audiovisual speech tasks and show that it can reduce the performance gap between the multimodal and corresponding unimodal models to a considerable extent. MUTUD can achieve this while reducing the model size and compute compared to multimodal models, in some cases by almost 80%.