NUAA-QMUL-AIIT at Memotion 3: Multi-modal Fusion with Squeeze-and-Excitation for Internet Meme Emotion Analysis

TL;DR

This paper introduces SEFusion, a novel multi-modal fusion technique for meme emotion analysis, which learns modality weights and improves classification performance in the Memotion 3 shared task.

Contribution

The paper presents SEFusion, a simple yet effective multi-modal fusion method that outperforms other systems in meme emotion classification tasks.

Findings

Ranked 1st on task A

Ranked 2nd on task C

Ranked 5th on task B

Abstract

This paper describes the participation of our NUAA-QMUL-AIIT team in the Memotion 3 shared task on meme emotion analysis. We propose a novel multi-modal fusion method, Squeeze-and-Excitation Fusion (SEFusion), and embed it into our system for emotion classification in memes. SEFusion is a simple fusion method that employs fully connected layers, reshaping, and matrix multiplication. SEFusion learns a weight for each modality and then applies it to its own modality feature. We evaluate the performance of our system on the three Memotion 3 sub-tasks. Among all participating systems in this Memotion 3 shared task, our system ranked first on task A, fifth on task B, and second on task C. Our proposed SEFusion provides the flexibility to fuse any features from different modalities. The source code for our method is published on https://github.com/xxxxxxxxy/memotion3-SEFusion.