Best Practices for Notification Studies for Security and Privacy Issues on the Internet

TL;DR

This paper provides comprehensive guidelines and best practices for conducting notification studies aimed at addressing security and privacy issues on the internet, emphasizing planning, automation, and reproducibility.

Contribution

It offers the first detailed set of documented pitfalls and best practices for notification studies, based on extensive experience and related research guidance.

Findings

Extensive planning improves study effectiveness.

Automation enhances efficiency and reproducibility.

Guidelines facilitate better data collection and interaction with recipients.

Abstract

Researchers help operators of vulnerable and non-compliant internet services by individually notifying them about security and privacy issues uncovered in their research. To improve efficiency and effectiveness of such efforts, dedicated notification studies are imperative. As of today, there is no comprehensive documentation of pitfalls and best practices for conducting such notification studies, which limits validity of results and impedes reproducibility. Drawing on our experience with such studies and guidance from related work, we present a set of guidelines and practical recommendations, including initial data collection, sending of notifications, interacting with the recipients, and publishing the results. We note that future studies can especially benefit from extensive planning and automation of crucial processes, i.e., activities that take place well before the first notifications are sent.