# Using Auxiliary Tasks In Multimodal Fusion Of Wav2vec 2.0 And BERT For Multimodal Emotion Recognition

## TL;DR

This paper enhances multimodal emotion recognition by integrating pretrained wav2vec 2.0 and BERT models with auxiliary tasks and a multi-head attention fusion mechanism, significantly improving accuracy on the IEMOCAP dataset.

## Contribution

It introduces auxiliary tasks to improve modality fusion and leverages pretrained models for better feature extraction in MER.

## Key findings

- Achieved 78.42% WA and 79.71% UA on IEMOCAP
- Improved over previous state-of-the-art models
- Demonstrated effectiveness of auxiliary tasks in multimodal fusion

## Abstract

The lack of data and the difficulty of multimodal fusion have always been challenges for multimodal emotion recognition (MER). In this paper, we propose to use pretrained models as upstream network, wav2vec 2.0 for audio modality and BERT for text modality, and finetune them in downstream task of MER to cope with the lack of data. For the difficulty of multimodal fusion, we use a K-layer multi-head attention mechanism as a downstream fusion module. Starting from the MER task itself, we design two auxiliary tasks to alleviate the insufficient fusion between modalities and guide the network to capture and align emotion-related features. Compared to the previous state-of-the-art models, we achieve a better performance by 78.42% Weighted Accuracy (WA) and 79.71% Unweighted Accuracy (UA) on the IEMOCAP dataset.