Does Task Complexity Moderate the Benefits of Liveness? A Controlled Experiment

TL;DR

This study investigates whether task complexity influences the effectiveness of live programming tools on debugging, finding that live tools generally reduce debugging time but the moderating role of task complexity remains inconclusive.

Contribution

It provides the first controlled experiment examining how task complexity affects the benefits of live introspection tools in debugging tasks.

Findings

Live tools significantly reduced debugging time.

No significant moderation effect of task complexity was found.

Participants' prior experience with liveness influences results.

Abstract

Live programming features can be found in a range of programming environments, from individual prototypes to widely used environments. While liveness is generally considered a useful property, there is little empirical evidence on when and how liveness can be beneficial. Even though there are few experimental studies, their results are largely inconclusive. We reviewed existing experiments and related studies to gather a collection of potential effects of liveness and moderating factors. Based on this collection, we concluded that **task complexity** and **prior experience addressing liveness** are potentially essential factors neglected in previous experiments. To fill this gap, we devised and conducted a controlled experiment (N = 37) testing the hypothesis that task complexity moderates the effects of live introspection tools on participants? debugging efficiency, given participants with prior experience with liveness. Our results do not support the hypothesis that task complexity moderates the effect of live introspection tools. This non-significant moderation effect might result from the low number of participants, as the data suggests a moderation effect. The results also show that in our experiment setting, live introspection tools significantly reduced the time participants took to debug the tasks. For researchers interested in the effects of liveness, our findings suggest that studies on liveness should make conscious choices about task complexity and participants' prior experience with liveness. For designers of programming environments, the results of our experiment are a step toward understanding when and how programming tools should be live to become more helpful to programmers.