Under the Bridge: Trolling and the Challenges of Recruiting Software Developers for Empirical Research Studies

TL;DR

This paper explores the unique challenges of recruiting software developers for empirical research, highlighting issues like participant recruitment, community engagement, and data poisoning, and suggests ways to address them.

Contribution

It identifies key human-factor challenges in software engineering research and offers a categorization and potential solutions based on pilot study insights.

Findings

Challenges in participant recruitment, community engagement, and data poisoning.

Categorization of issues faced in human-subjects software engineering research.

Potential benefits of addressing these challenges for future studies.

Abstract

Much of software engineering research focuses on tools, algorithms, and optimization of software. Recently, we, as a community, have come to acknowledge that there is a gap in meta-research and addressing the human-factors in software engineering research. Through meta research, we aim to deepen our understanding of online participant recruitment and human-subjects software engineering research. In this paper we motivate the need to consider the unique challenges that human studies pose in software engineering research. We present several challenges faced by our research team in several distinct research studies, how they affected research, and motivate how, as researchers, we can address these challenges. We present results from a pilot study and categorize issues faced into three broad categories including participant recruitment, community engagement, and data poisoning. We further discuss how we can address these challenges and outline the benefits a full-study could provide to the software engineering research community.