Training Strategies to Handle Missing Modalities for Audio-Visual Expression Recognition

TL;DR

This paper investigates training strategies for audio-visual expression recognition systems to maintain performance when one modality is missing, proposing methods that improve robustness in real-world scenarios.

Contribution

It introduces a training approach that randomly ablates visual inputs during training, enhancing model robustness to missing modalities in audio-visual expression recognition.

Findings

Models trained with ablation strategies show up to 17% improvement in performance.

Proposed methods improve generalization in real-world scenarios with missing cues.

Significant gains observed on in-the-wild datasets.

Abstract

Automatic audio-visual expression recognition can play an important role in communication services such as tele-health, VOIP calls and human-machine interaction. Accuracy of audio-visual expression recognition could benefit from the interplay between the two modalities. However, most audio-visual expression recognition systems, trained in ideal conditions, fail to generalize in real world scenarios where either the audio or visual modality could be missing due to a number of reasons such as limited bandwidth, interactors' orientation, caller initiated muting. This paper studies the performance of a state-of-the art transformer when one of the modalities is missing. We conduct ablation studies to evaluate the model in the absence of either modality. Further, we propose a strategy to randomly ablate visual inputs during training at the clip or frame level to mimic real world scenarios. Results conducted on in-the-wild data, indicate significant generalization in proposed models trained on missing cues, with gains up to 17% for frame level ablations, showing that these training strategies cope better with the loss of input modalities.