Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognition

TL;DR

This paper introduces a multimodal emotion recognition approach combining pre-trained wav2vec 2.0 and BERT models with multi-level fusion and multi-granularity features, achieving improved accuracy on IEMOCAP.

Contribution

It proposes a novel multi-granularity framework and fusion strategies leveraging pre-trained models for enhanced multimodal emotion recognition.

Findings

Outperforms baseline by 1.3% UA on IEMOCAP

Utilizes multi-level fusion with coattention and late fusion

Employs multi-granularity speech embeddings

Abstract

The research and applications of multimodal emotion recognition have become increasingly popular recently. However, multimodal emotion recognition faces the challenge of lack of data. To solve this problem, we propose to use transfer learning which leverages state-of-the-art pre-trained models including wav2vec 2.0 and BERT for this task. Multi-level fusion approaches including coattention-based early fusion and late fusion with the models trained on both embeddings are explored. Also, a multi-granularity framework which extracts not only frame-level speech embeddings but also segment-level embeddings including phone, syllable and word-level speech embeddings is proposed to further boost the performance. By combining our coattention-based early fusion model and late fusion model with the multi-granularity feature extraction framework, we obtain result that outperforms best baseline approaches by 1.3% unweighted accuracy (UA) on the IEMOCAP dataset.