Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution

TL;DR

This paper presents a winning multimodal transformer approach for hateful meme detection, integrating external labels and in-domain pretraining to enhance visual-linguistic understanding, achieving top performance in a recent challenge.

Contribution

It introduces a novel extension of visual-linguistic transformers with external labels and domain-specific pretraining, advancing the state-of-the-art in hateful meme detection.

Findings

Achieved first place in the Hateful Meme Detection Challenge 2020

Enhanced transformer performance with external label integration

Identified limitations and future directions for the methodology

Abstract

Hateful meme detection is a new research area recently brought out that requires both visual, linguistic understanding of the meme and some background knowledge to performing well on the task. This technical report summarises the first place solution of the Hateful Meme Detection Challenge 2020, which extending state-of-the-art visual-linguistic transformers to tackle this problem. At the end of the report, we also point out the shortcomings and possible directions for improving the current methodology.