Evaluating Language Models on Grooming Risk Estimation Using Fuzzy Theory

TL;DR

This paper investigates the ability of SBERT language models to assess grooming risk in conversations, highlighting challenges in detecting indirect speech and the need for more robust modeling in high-risk domains.

Contribution

The study evaluates SBERT's effectiveness in discerning grooming risk levels and identifies limitations in handling indirect speech and implicit language.

Findings

Fine-tuning improves grooming score predictions.

High variance in predictions for high-risk contexts.

Errors often occur with indirect speech and non-explicit content.

Abstract

Encoding implicit language presents a challenge for language models, especially in high-risk domains where maintaining high precision is important. Automated detection of online child grooming is one such critical domain, where predators manipulate victims using a combination of explicit and implicit language to convey harmful intentions. While recent studies have shown the potential of Transformer language models like SBERT for preemptive grooming detection, they primarily depend on surface-level features and approximate real victim grooming processes using vigilante and law enforcement conversations. The question of whether these features and approximations are reasonable has not been addressed thus far. In this paper, we address this gap and study whether SBERT can effectively discern varying degrees of grooming risk inherent in conversations, and evaluate its results across different participant groups. Our analysis reveals that while fine-tuning aids language models in learning to assign grooming scores, they show high variance in predictions, especially for contexts containing higher degrees of grooming risk. These errors appear in cases that 1) utilize indirect speech pathways to manipulate victims and 2) lack sexually explicit content. This finding underscores the necessity for robust modeling of indirect speech acts by language models, particularly those employed by predators.