Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learning

TL;DR

This paper introduces a shared representation learning approach for harmful meme detection that remains effective even when one modality, like text, is missing, addressing real-world data incompleteness issues.

Contribution

It proposes a novel shared representation learning method that improves robustness of harmful meme detection under missing modalities, a first in this research area.

Findings

Outperforms existing methods when text is missing

Enhances visual feature integration and robustness

Facilitates real-world application of harmful meme detection

Abstract

Internet memes are powerful tools for communication, capable of spreading political, psychological, and sociocultural ideas. However, they can be harmful and can be used to disseminate hate toward targeted individuals or groups. Although previous studies have focused on designing new detection methods, these often rely on modal-complete data, such as text and images. In real-world settings, however, modalities like text may be missing due to issues like poor OCR quality, making existing methods sensitive to missing information and leading to performance deterioration. To address this gap, in this paper, we present the first-of-its-kind work to comprehensively investigate the behavior of harmful meme detection methods in the presence of modal-incomplete data. Specifically, we propose a new baseline method that learns a shared representation for multiple modalities by projecting them independently. These shared representations can then be leveraged when data is modal-incomplete. Experimental results on two benchmark datasets demonstrate that our method outperforms existing approaches when text is missing. Moreover, these results suggest that our method allows for better integration of visual features, reducing dependence on text and improving robustness in scenarios where textual information is missing. Our work represents a significant step forward in enabling the real-world application of harmful meme detection, particularly in situations where a modality is absent.