Unimodal Intermediate Training for Multimodal Meme Sentiment Classification

TL;DR

This paper introduces a novel unimodal intermediate training approach that enhances multimodal meme sentiment classification by leveraging abundant unimodal data, significantly improving performance and reducing the need for labelled memes.

Contribution

It proposes a new supervised intermediate training method using unimodal data to improve multimodal meme sentiment classification, addressing data scarcity issues.

Findings

Significant performance improvement with unimodal text data.

Can reduce labelled meme data by 40% without performance loss.

Statistically significant results demonstrating effectiveness.

Abstract

Internet Memes remain a challenging form of user-generated content for automated sentiment classification. The availability of labelled memes is a barrier to developing sentiment classifiers of multimodal memes. To address the shortage of labelled memes, we propose to supplement the training of a multimodal meme classifier with unimodal (image-only and text-only) data. In this work, we present a novel variant of supervised intermediate training that uses relatively abundant sentiment-labelled unimodal data. Our results show a statistically significant performance improvement from the incorporation of unimodal text data. Furthermore, we show that the training set of labelled memes can be reduced by 40% without reducing the performance of the downstream model.