40 Years of Designing Code Comprehension Experiments: A Systematic Mapping Study

TL;DR

This systematic mapping study reviews 95 code comprehension experiments over 40 years, highlighting common design choices, deficiencies, and providing guidance for future research in experimental design.

Contribution

It offers a structured overview of design characteristics in code comprehension experiments, aiding researchers in designing rigorous and comparable studies.

Findings

Identified prevalent topics and design options in code comprehension studies

Highlighted common deficiencies in study designs

Provided guidance for designing future experiments

Abstract

The relevance of code comprehension in a developer's daily work was recognized more than 40 years ago. Consequently, many experiments were conducted to find out how developers could be supported during code comprehension and which code characteristics contribute to better comprehension. Today, such studies are more common than ever. While this is great for advancing the field, the number of publications makes it difficult to keep an overview. Additionally, designing rigorous code comprehension experiments with human participants is a challenging task, and the multitude of design options can make it difficult for researchers, especially newcomers to the field, to select a suitable design. We therefore conducted a systematic mapping study of 95 source code comprehension experiments published between 1979 and 2019. By structuring the design characteristics of code comprehension studies, we provide a basis for subsequent discussion of the huge diversity of design options in the face of a lack of basic research on their consequences and comparability. We describe what topics have been studied, as well as how these studies have been designed, conducted, and reported. Frequently chosen design options and deficiencies are pointed out to support researchers of all levels of domain expertise in designing their own studies.