Codec at SemEval-2022 Task 5: Multi-Modal Multi-Transformer Misogynous Meme Classification Framework

TL;DR

This paper presents a multi-modal, multi-label classification framework for misogyny detection in memes, combining state-of-the-art architectures, multi-task learning, and multiple objectives to improve performance in a resource-efficient manner.

Contribution

It introduces a generic multi-modal embedding and classification framework that leverages multi-task learning and multiple objectives, advancing misogyny detection in multimedia content.

Findings

Effective multi-modal embedding capturing diverse semantic signals

Improved performance through multi-task learning with multiple datasets

Enhanced model robustness via multi-objective regularization

Abstract

In this paper we describe our work towards building a generic framework for both multi-modal embedding and multi-label binary classification tasks, while participating in task 5 (Multimedia Automatic Misogyny Identification) of SemEval 2022 competition. Since pretraining deep models from scratch is a resource and data hungry task, our approach is based on three main strategies. We combine different state-of-the-art architectures to capture a wide spectrum of semantic signals from the multi-modal input. We employ a multi-task learning scheme to be able to use multiple datasets from the same knowledge domain to help increase the model's performance. We also use multiple objectives to regularize and fine tune different system components.