Further Evidence on a Controversial Topic about Human-Based Experiments: Professionals vs. Students

TL;DR

This study compares students and professionals in bug-fixing tasks, revealing students outperform professionals despite more realistic conditions for the latter, thus questioning the external validity of student-based experiments in software engineering.

Contribution

It provides new empirical evidence contrasting previous findings by comparing students and professionals under different experimental conditions in SE tasks.

Findings

Students outperformed professionals in bug-fixing.

More realistic conditions did not improve professional performance.

Results challenge assumptions about external validity of student experiments.

Abstract

Most Software Engineering (SE) human-based controlled experiments rely on students as participants, raising concerns about their external validity. Specifically, the realism of results obtained from students and their applicability to the software industry remains in question. In this short paper, we bring further evidence on this controversial point. To do so, we compare 62 students and 42 software professionals on a bug-fixing task on the same Java program. The students were enrolled in a Bachelor's program in Computer Science, while the professionals were employed by two multinational companies (for one of them, the professionals were from two offices). Some variations in the experimental settings of the two groups (students and professionals) were present. For instance, the experimental environment of the experiment with professionals was more realistic; i.e., they faced some stress factors such as interruptions during the bug-fixing task. Considering the differences between the two groups of participants, the gathered data show that the students outperformed the professionals in fixing bugs. This diverges to some extent from past empirical evidence. Rather than presenting definitive conclusions, our results aim to catalyze the discussion on the use of students in experiments and pave the way for future investigations. Specifically, our results encourage us to examine the complex factors influencing SE tasks, making experiments as more realistic as possible.