UPB at SemEval-2020 Task 8: Joint Textual and Visual Modeling in a Multi-Task Learning Architecture for Memotion Analysis

TL;DR

This paper presents a multimodal multi-task learning system combining textual and visual data for Memotion analysis, achieving top rankings in the SemEval-2020 competition.

Contribution

It introduces a novel joint architecture using ALBERT and VGG-16 for effective meme analysis, surpassing baseline performance in all subtasks.

Findings

Achieved 1st place in Subtask B with 0.5183 macro F1-score.

Outperformed baseline results significantly across all subtasks.

Demonstrated the effectiveness of multimodal multi-task learning for meme sentiment analysis.

Abstract

Users from the online environment can create different ways of expressing their thoughts, opinions, or conception of amusement. Internet memes were created specifically for these situations. Their main purpose is to transmit ideas by using combinations of images and texts such that they will create a certain state for the receptor, depending on the message the meme has to send. These posts can be related to various situations or events, thus adding a funny side to any circumstance our world is situated in. In this paper, we describe the system developed by our team for SemEval-2020 Task 8: Memotion Analysis. More specifically, we introduce a novel system to analyze these posts, a multimodal multi-task learning architecture that combines ALBERT for text encoding with VGG-16 for image representation. In this manner, we show that the information behind them can be properly revealed. Our approach achieves good performance on each of the three subtasks of the current competition, ranking 11th for Subtask A (0.3453 macro F1-score), 1st for Subtask B (0.5183 macro F1-score), and 3rd for Subtask C (0.3171 macro F1-score) while exceeding the official baseline results by high margins.