Worse Than Spam: Issues In Sampling Software Developers

TL;DR

This paper examines the challenges and ethical issues in sampling software developers for surveys, highlighting effective strategies and advocating for ethical guidelines to improve external validity in empirical software engineering research.

Contribution

It provides an analysis of sampling strategies, ethical considerations, and proposes maintaining demographic data to enhance research validity in software engineering.

Findings

Public media sampling is most effective and efficient.

Ethical issues vary across sampling strategies.

A guideline for ethical sampling is proposed.

Abstract

Background: Reaching out to professional software developers is a crucial part of empirical software engineering research. One important method to investigate the state of practice is survey research. As drawing a random sample of professional software developers for a survey is rarely possible, researchers rely on various sampling strategies. Objective: In this paper, we report on our experience with different sampling strategies we employed, highlight ethical issues, and motivate the need to maintain a collection of key demographics about software developers to ease the assessment of the external validity of studies. Method: Our report is based on data from two studies we conducted in the past. Results: Contacting developers over public media proved to be the most effective and efficient sampling strategy. However, we not only describe the perspective of researchers who are interested in reaching goals like a large number of participants or a high response rate, but we also shed light onto ethical implications of different sampling strategies. We present one specific ethical guideline and point to debates in other research communities to start a discussion in the software engineering research community about which sampling strategies should be considered ethical.