Countering Malicious Content Moderation Evasion in Online Social Networks: Simulation and Detection of Word Camouflage

TL;DR

This paper introduces a multilingual Transformer-based approach and a simulation tool to detect and generate camouflaged malicious content in social networks, enhancing moderation capabilities against evolving evasion tactics.

Contribution

It presents a novel multilingual detection model and a simulation tool for content evasion techniques, addressing the challenge of linguistic camouflage in online moderation.

Findings

The multilingual NER model achieved an overall weighted F1 score of 0.8795.

The simulation tool 'pyleetspeak' effectively generates camouflaged content for testing.

The approach improves detection of diverse and mixed camouflage techniques.

Abstract

Content moderation is the process of screening and monitoring user-generated content online. It plays a crucial role in stopping content resulting from unacceptable behaviors such as hate speech, harassment, violence against specific groups, terrorism, racism, xenophobia, homophobia, or misogyny, to mention some few, in Online Social Platforms. These platforms make use of a plethora of tools to detect and manage malicious information; however, malicious actors also improve their skills, developing strategies to surpass these barriers and continuing to spread misleading information. Twisting and camouflaging keywords are among the most used techniques to evade platform content moderation systems. In response to this recent ongoing issue, this paper presents an innovative approach to address this linguistic trend in social networks through the simulation of different content evasion techniques and a multilingual Transformer model for content evasion detection. In this way, we share with the rest of the scientific community a multilingual public tool, named "pyleetspeak" to generate/simulate in a customizable way the phenomenon of content evasion through automatic word camouflage and a multilingual Named-Entity Recognition (NER) Transformer-based model tuned for its recognition and detection. The multilingual NER model is evaluated in different textual scenarios, detecting different types and mixtures of camouflage techniques, achieving an overall weighted F1 score of 0.8795. This article contributes significantly to countering malicious information by developing multilingual tools to simulate and detect new methods of evasion of content on social networks, making the fight against information disorders more effective.