The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderation

TL;DR

This paper investigates how emojis contribute to harmful online messages on Twitter, analyzing their roles and proposing an LLM-based moderation method that reduces offensiveness while maintaining message intent.

Contribution

It introduces a systematic analysis of emoji usage in offensive content and presents a novel LLM-powered moderation pipeline for selective emoji replacement.

Findings

Emojis can acquire harmful meanings through context and symbolism.

The proposed moderation pipeline effectively reduces perceived offensiveness.

Emoji effects vary across different types of offensive content.

Abstract

Social media platforms have become central to modern communication, yet they also harbor offensive content that challenges platform safety and inclusivity. While prior research has primarily focused on textual indicators of offense, the role of emojis, ubiquitous visual elements in online discourse, remains underexplored. Emojis, despite being rarely offensive in isolation, can acquire harmful meanings through symbolic associations, sarcasm, and contextual misuse. In this work, we systematically examine emoji contributions to offensive Twitter messages, analyzing their distribution across offense categories and how users exploit emoji ambiguity. To address this, we propose an LLM-powered, multi-step moderation pipeline that selectively replaces harmful emojis while preserving the tweet's semantic intent. Human evaluations confirm our approach effectively reduces perceived offensiveness without sacrificing meaning. Our analysis also reveals heterogeneous effects across offense types, offering nuanced insights for online communication and emoji moderation.