A Fuzzy Evaluation of Sentence Encoders on Grooming Risk Classification

TL;DR

This study evaluates sentence encoders for grooming risk classification in online chats, revealing that models struggle with indirect and coded language, emphasizing the need for more robust detection methods in social media safety.

Contribution

The paper introduces a fuzzy evaluation framework to assess how well models align with human perceptions of grooming, especially in coded language scenarios, highlighting limitations of current fine-tuned transformers.

Findings

Models fail to detect indirect and coded grooming language

High out-of-vocabulary words correlate with misclassification

Current models need improvement for noisy chat inputs

Abstract

With the advent of social media, children are becoming increasingly vulnerable to the risk of grooming in online settings. Detecting grooming instances in an online conversation poses a significant challenge as the interactions are not necessarily sexually explicit, since the predators take time to build trust and a relationship with their victim. Moreover, predators evade detection using indirect and coded language. While previous studies have fine-tuned Transformers to automatically identify grooming in chat conversations, they overlook the impact of coded and indirect language on model predictions, and how these align with human perceptions of grooming. In this paper, we address this gap and evaluate bi-encoders on the task of classifying different degrees of grooming risk in chat contexts, for three different participant groups, i.e. law enforcement officers, real victims, and decoys. Using a fuzzy-theoretic framework, we map human assessments of grooming behaviors to estimate the actual degree of grooming risk. Our analysis reveals that fine-tuned models fail to tag instances where the predator uses indirect speech pathways and coded language to evade detection. Further, we find that such instances are characterized by a higher presence of out-of-vocabulary (OOV) words in samples, causing the model to misclassify. Our findings highlight the need for more robust models to identify coded language from noisy chat inputs in grooming contexts.