Large Language Models in the Abuse Detection Pipeline

TL;DR

This survey explores how Large Language Models are transforming the abuse detection lifecycle by enhancing contextual understanding, policy interpretation, and governance, while addressing deployment challenges.

Contribution

It provides a comprehensive lifecycle-oriented analysis of LLM integration into abuse detection, highlighting research, practices, and architectural considerations.

Findings

LLMs improve contextual reasoning in abuse detection.

They support policy interpretation and explanation generation.

Challenges include latency, cost, robustness, and fairness.

Abstract

Online abuse has grown increasingly complex, spanning toxic language, harassment, manipulation, and fraudulent behavior. Traditional machine-learning approaches dependent on static classifiers and labor-intensive labeling struggle to keep pace with evolving threat patterns and nuanced policy requirements. Large Language Models introduce new capabilities for contextual reasoning, policy interpretation, explanation generation, and cross-modal understanding, enabling them to support multiple stages of modern safety systems. This survey provides a lifecycle-oriented analysis of how LLMs are being integrated into the Abuse Detection Lifecycle (ADL), which we define across four stages: (I) Label \& Feature Generation, (II) Detection, (III) Review \& Appeals, and (IV) Auditing \& Governance. For each stage, we synthesize emerging research and industry practices, highlight architectural considerations for production deployment, and examine the strengths and limitations of LLM-driven approaches. We conclude by outlining key challenges including latency, cost-efficiency, determinism, adversarial robustness, and fairness and discuss future research directions needed to operationalize LLMs as reliable, accountable components of large-scale abuse-detection and governance systems.