'Show It, Don't Just Say It': The Complementary Effects of Instruction Multimodality for Software Guidance

TL;DR

This study investigates how different instructional modalities like speech, annotations, and remote control affect software learning, highlighting their complementary roles and the trade-offs between guidance precision and learner agency.

Contribution

It provides empirical insights into how multimodal instructional channels are used and balanced in software tutoring, introducing new considerations of digital territoriality and agency.

Findings

Annotations add spatial precision to speech instructions.

Remote control enhances spatial and temporal guidance.

Both modalities can intrude on learner agency.

Abstract

Designing adaptive tutoring systems for software learning presents challenges in determining appropriate instructional modalities. To inform the design of such systems, we conducted an observational study of ten human teacher-student pairs (N=10), where experienced design software users taught novices two new graphic design software features through multi-step procedures. These lessons were limited to three communication channels (speech, visual annotations, and remote screen control) to mimic possible AI tutor modalities. We found that annotations complement speech with spatial precision and remote control complements it with spatial and temporal precision, but both cause intrusion to learner agency. Teachers adaptively select modalities to balance the need for instruction progress with students' cognitive engagement and sense of digital territory ownership. Our results provide further support to the contiguity principles and the value of agency in learning, while suggesting precision-agency trade-off and digital territoriality as new design constraints for adaptive software guidance.