hate-alert@DravidianLangTech-ACL2022: Ensembling Multi-Modalities for Tamil TrollMeme Classification

TL;DR

This paper presents an ensemble approach combining text and image models to detect Tamil troll memes, achieving improved classification performance and ranking second in a shared task.

Contribution

It introduces a multi-modal fusion model for Tamil troll meme detection, addressing the lack of benchmark datasets and models for multilingual meme analysis.

Findings

Fusion of text and image models improves classification accuracy.

Text-based model outperforms image-based model for non-troll memes.

Achieved a 0.561 weighted F1 score, ranking second in the task.

Abstract

Social media platforms often act as breeding grounds for various forms of trolling or malicious content targeting users or communities. One way of trolling users is by creating memes, which in most cases unites an image with a short piece of text embedded on top of it. The situation is more complex for multilingual(e.g., Tamil) memes due to the lack of benchmark datasets and models. We explore several models to detect Troll memes in Tamil based on the shared task, "Troll Meme Classification in DravidianLangTech2022" at ACL-2022. We observe while the text-based model MURIL performs better for Non-troll meme classification, the image-based model VGG16 performs better for Troll-meme classification. Further fusing these two modalities help us achieve stable outcomes in both classes. Our fusion model achieved a 0.561 weighted average F1 score and ranked second in this task.