Caption Enriched Samples for Improving Hateful Memes Detection

TL;DR

This paper enhances hateful memes detection by integrating automatically generated image captions, which improves model performance, especially when combined with continued pre-training of language models on captioned data.

Contribution

It introduces the use of off-the-shelf image captioning to enrich samples and demonstrates benefits of continued pre-training on captioned data for better detection accuracy.

Findings

Automatic captions improve model performance.

Continued pre-training on captioned data boosts accuracy.

Multimodal and unimodal models benefit from caption integration.

Abstract

The recently introduced hateful meme challenge demonstrates the difficulty of determining whether a meme is hateful or not. Specifically, both unimodal language models and multimodal vision-language models cannot reach the human level of performance. Motivated by the need to model the contrast between the image content and the overlayed text, we suggest applying an off-the-shelf image captioning tool in order to capture the first. We demonstrate that the incorporation of such automatic captions during fine-tuning improves the results for various unimodal and multimodal models. Moreover, in the unimodal case, continuing the pre-training of language models on augmented and original caption pairs, is highly beneficial to the classification accuracy.