The Price of Using Students
Dag I.K. Sj{\o}berg, Gunnar R. Bergersen

TL;DR
This paper discusses the advantages and disadvantages of using students versus professionals as subjects in software engineering experiments, emphasizing the importance of understanding skill differences to improve research validity.
Contribution
It offers insights and suggestions for improving experimental design involving students and addresses misconceptions about using professionals in software engineering research.
Findings
Skill differences impact experimental outcomes.
Using students can be valid with proper controls.
Professional subjects have their own drawbacks.
Abstract
In a recent article, Falessi et al. (2017) call for a deeper understanding of the pros and cons of using students and professionals in experiments. The authors state: we have observed too many times that our papers were rejected because we used students as subjects. Good experiments with students are certainly a valuable asset in the body of research in software engineering. Papers should thus not be rejected solely on the ground that the subjects are students. However, the distribution in skill is different for students and professionals. Since previous studies have shown that skill may have a moderating effect on the treatment of participants, we are concerned that studies involving developers with only low to medium skill (i.e., students) may result in wrong inferences about which technology, method or tool is better in the software industry. We therefore provide suggestions for how…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Engineering Research · Software Engineering Techniques and Practices · Open Source Software Innovations
