"Is Reporting Worth the Sacrifice of Revealing What I Have Sent?": Privacy Considerations When Reporting on End-to-End Encrypted Platforms

TL;DR

This study explores user privacy concerns and mental models regarding reporting mechanisms on end-to-end encrypted messaging platforms, highlighting trust issues and privacy risks that influence reporting behavior and suggesting design improvements.

Contribution

It provides empirical insights into user perceptions and privacy considerations in reporting on E2EE platforms, informing privacy-preserving system design.

Findings

Users expect platforms to store detailed reporting data, risking privacy.

Users trust platform moderators more than community moderators.

Reporting decisions are influenced by perceived effectiveness and privacy concerns.

Abstract

User reporting is an essential component of content moderation on many online platforms -- in particular, on end-to-end encrypted (E2EE) messaging platforms where platform operators cannot proactively inspect message contents. However, users' privacy concerns when considering reporting may impede the effectiveness of this strategy in regulating online harassment. In this paper, we conduct interviews with 16 users of E2EE platforms to understand users' mental models of how reporting works and their resultant privacy concerns and considerations surrounding reporting. We find that users expect platforms to store rich longitudinal reporting datasets, recognizing both their promise for better abuse mitigation and the privacy risk that platforms may exploit or fail to protect them. We also find that users have preconceptions about the respective capabilities and risks of moderators at the platform versus community level -- for instance, users trust platform moderators more to not abuse their power but think community moderators have more time to attend to reports. These considerations, along with perceived effectiveness of reporting and how to provide sufficient evidence while maintaining privacy, shape how users decide whether, to whom, and how much to report. We conclude with design implications for a more privacy-preserving reporting system on E2EE messaging platforms.