Not All Scaffolds Are Equal: How Initiation Mode Determines EMME Effectiveness in Debugging

TL;DR

This study examines how the timing and initiation mode of eye movement modeling examples (EMME) influence their effectiveness in aiding novice programmers during debugging tasks, highlighting the importance of adaptive scaffold delivery.

Contribution

It demonstrates that human-mediated initiation of EMME yields better performance and engagement than automated triggering based on physiological indicators, emphasizing the need for careful design of scaffold timing.

Findings

All EMME conditions improved debugging performance over control.

Human-initiated EMME outperformed automated triggering in effectiveness.

Automated triggering based on low pupillary effort caused disruptive behaviors.

Abstract

Adaptive learning technologies increasingly rely on real time physiological analytics to trigger instructional support automatically yet how system driven decisions interact with learners ongoing problem solving processes remains poorly understood. Eye Movement Modeling Examples have shown promise as attention guidance tools but have been studied predominantly as static instructional materials rather than as adaptive scaffolds whose timing and initiation control can vary. This study investigates whether scaffold initiation mode shapes EMME effectiveness in novice programmers debugging and specifically whether automated triggering based on a single physiological indicator of low mental effort is a viable basis for adaptive scaffold delivery. A between subjects experiment was conducted with 120 undergraduate computer science students randomly assigned to one of four conditions: teacher initiated, learner initiated, automated or no scaffold control. Participants completed ten Python debugging tasks while eye tracking data, video interaction logs and performance scores were recorded. All EMME conditions outperformed the control. However human mediated initiation whether teacher or learner consistently produced higher performance than automated triggering and more integrative engagement with the EMME material. Automated triggering based on sustained low pupillary activity was associated with disruptive behavioral patterns suggesting mistimed delivery. EMME also eliminated the performance advantage of prior programming knowledge across all initiation modes. These findings establish scaffold initiation timing and control as critical design variables for EMME and adaptive learning technologies more broadly and demonstrate that a single low effort physiological threshold is insufficient as a trigger criterion for complex problem solving support.