Weapons of Online Harassment: Menacing and Profiling Users via Social Apps

TL;DR

This paper investigates online harassment in social apps by analyzing over 3 million reviews, identifying prevalent harassment forms, and developing a model to detect and notify about harassment-enabled apps.

Contribution

It introduces a large dataset of app reviews, identifies two main harassment types, and creates a high-accuracy model for detecting harassment in social apps.

Findings

Menacing and Profiling are the main harassment forms.

High recall rates of 90% for Menacing and 85% for Profiling.

Identified 1,395 apps enabling harassment.

Abstract

Viewing social apps as sociotechnical systems makes clear that they are not mere pieces of technology but mediate human interaction and may unintentionally enable harmful behaviors like online harassment. As more users interact through social apps, instances of harassment increase. We observed that app reviews often describe harassment. Accordingly, we built a dataset of over 3 million reviews and 1,800 apps. We discovered that two forms of harassment are prevalent, Menacing and Profiling. We built a computational model for identifying reviews indicating harassment, achieving high recalls of 90% for Menacing and 85% for Profiling. We analyzed the data further to better understand the terrain of harassment. Surprisingly, abusers most often have female identities. Also, what distinguishes negative from neutral reviews is the greater prevalence of anger, disgust, and fear. Applying our model, we identified 1,395 apps enabling harassment and notified developers of the top 48 with the highest user-reported harassment.