A vector quantized masked autoencoder for audiovisual speech emotion recognition

TL;DR

This paper introduces VQ-MAE-AV, a self-supervised multimodal model that learns audiovisual speech representations from unlabeled data using masked autoencoders and vector quantization, improving emotion recognition.

Contribution

The paper presents a novel self-supervised framework combining vector quantized autoencoders and masked autoencoders for audiovisual speech emotion recognition.

Findings

Achieves state-of-the-art results on multiple emotion recognition datasets.

Effectively leverages unlabeled audiovisual speech data for representation learning.

Improves robustness in both controlled and in-the-wild conditions.

Abstract

An important challenge in emotion recognition is to develop methods that can leverage unlabeled training data. In this paper, we propose the VQ-MAE-AV model, a self-supervised multimodal model that leverages masked autoencoders to learn representations of audiovisual speech without labels. The model includes vector quantized variational autoencoders that compress raw audio and visual speech data into discrete tokens. The audiovisual speech tokens are used to train a multimodal masked autoencoder that consists of an encoder-decoder architecture with attention mechanisms. The model is designed to extract both local (i.e., at the frame level) and global (i.e., at the sequence level) representations of audiovisual speech. During self-supervised pre-training, the VQ-MAE-AV model is trained on a large-scale unlabeled dataset of audiovisual speech, for the task of reconstructing randomly masked audiovisual speech tokens and with a contrastive learning strategy. During this pre-training, the encoder learns to extract a representation of audiovisual speech that can be subsequently leveraged for emotion recognition. During the supervised fine-tuning stage, a small classification model is trained on top of the VQ-MAE-AV encoder for an emotion recognition task. The proposed approach achieves state-of-the-art emotion recognition results across several datasets in both controlled and in-the-wild conditions.