Anchoring Code Understandability Evaluations Through Task Descriptions

TL;DR

This study reveals that initial scenario descriptions in code comprehension experiments significantly influence participants' perceived difficulty of code, highlighting the need for more robust evaluation methods to avoid cognitive biases.

Contribution

It demonstrates the impact of anchoring effects on subjective code comprehensibility ratings and discusses experimental design considerations to improve validity in code comprehension research.

Findings

Participants' difficulty ratings are influenced by initial scenario hints.

Hard hints lead to higher perceived difficulty across all participant groups.

Recommends using more robust measures to mitigate anchoring bias.

Abstract

In code comprehension experiments, participants are usually told at the beginning what kind of code comprehension task to expect. Describing experiment scenarios and experimental tasks will influence participants in ways that are sometimes hard to predict and control. In particular, describing or even mentioning the difficulty of a code comprehension task might anchor participants and their perception of the task itself. In this study, we investigated in a randomized, controlled experiment with 256 participants (50 software professionals and 206 computer science students) whether a hint about the difficulty of the code to be understood in a task description anchors participants in their own code comprehensibility ratings. Subjective code evaluations are a commonly used measure for how well a developer in a code comprehension study understood code. Accordingly, it is important to understand how robust these measures are to cognitive biases such as the anchoring effect. Our results show that participants are significantly influenced by the initial scenario description in their assessment of code comprehensibility. An initial hint of hard to understand code leads participants to assess the code as harder to understand than participants who received no hint or a hint of easy to understand code. This affects students and professionals alike. We discuss examples of design decisions and contextual factors in the conduct of code comprehension experiments that can induce an anchoring effect, and recommend the use of more robust comprehension measures in code comprehension studies to enhance the validity of results.