Considerations and Pitfalls in Controlled Experiments on Code Comprehension

TL;DR

This paper discusses the complexities and potential pitfalls in designing controlled experiments for studying code comprehension, emphasizing the importance of sound methodology across various experimental factors.

Contribution

It provides a comprehensive analysis of common issues and considerations in conducting controlled experiments on code comprehension to improve research validity.

Findings

Identifies key factors influencing experiment outcomes

Highlights common methodological pitfalls

Offers guidelines for sound experimental design

Abstract

Understanding program code is a complicated endeavor. As such, myriad different factors can influence the outcome. Investigations of program comprehension, and in particular those using controlled experiments, have to take these factors into account. In order to promote the development and use of sound experimental methodology, we discuss potential problems with regard to the experimental subjects, the code they work on, the tasks they are asked to perform, and the metrics for their performance.