Context-Aware Detection and Victim-Centered Response Generation for Online Harassment in Private Messaging
Pinxian Lu, Nimra Ishfaq, Emma Win, Morgan Rose, Sierra R Strickland, Candice L Biernesser, Jamie Zelazny, Munmun De Choudhury

TL;DR
This paper explores how large language models can detect and respond to online harassment in private messages, emphasizing context-awareness and victim support, based on a new dataset of Instagram direct messages.
Contribution
It introduces a context-aware classification pipeline and a victim-centered response framework for harassment detection and support in private messaging, outperforming existing public data-based models.
Findings
The classification pipeline outperforms baseline toxicity classifiers.
AI-generated responses are perceived as more helpful than original responses.
Human evaluators rated AI responses as significantly more emotionally supportive.
Abstract
Online harassment is a widespread social and public health concern, yet most computational approaches for detecting and addressing harassment focus on publicly visible social media content rather than private messaging environments. Private conversations present unique challenges because harmful interactions often unfold through context-dependent, multi-turn exchanges, while victims may lack timely support during moments of harassment. In this study, we investigate how large language models (LLMs) can support both the detection of and response to online harassment in private messaging. Using a dataset of 80,053 Instagram direct messages donated by 26 adolescents aged 12-18, including youth with suicide risk factors, we first construct a human-labeled dataset of online harassment in private conversations and develop a context-aware cascading LLM classification pipeline. The proposed…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHate Speech and Cyberbullying Detection · Authorship Attribution and Profiling · Spam and Phishing Detection
