The Hateful Memes Challenge Next Move

TL;DR

This paper explores semi-supervised learning to improve classification of hateful memes, but finds limited gains from adding new data and highlights challenges in labeling unlabeled memes.

Contribution

It introduces a semi-supervised approach for classifying hateful memes and discusses the difficulties in labeling unlabeled meme data.

Findings

Semi-supervised learning requires human filtering of unlabeled data.

Adding limited new data does not improve classification accuracy.

Challenges in labeling unlabeled memes hinder performance gains.

Abstract

State-of-the-art image and text classification models, such as Convolutional Neural Networks and Transformers, have long been able to classify their respective unimodal reasoning satisfactorily with accuracy close to or exceeding human accuracy. However, images embedded with text, such as hateful memes, are hard to classify using unimodal reasoning when difficult examples, such as benign confounders, are incorporated into the data set. We attempt to generate more labeled memes in addition to the Hateful Memes data set from Facebook AI, based on the framework of a winning team from the Hateful Meme Challenge. To increase the number of labeled memes, we explore semi-supervised learning using pseudo-labels for newly introduced, unlabeled memes gathered from the Memotion Dataset 7K. We find that the semi-supervised learning task on unlabeled data required human intervention and filtering and that adding a limited amount of new data yields no extra classification performance.