What Is the Best Way For Developers to Learn New Software Tools? An Empirical Comparison Between a Text and a Video Tutorial

TL;DR

This study empirically compares text and video tutorials for teaching software tools, finding minimal efficiency differences and highlighting user preferences, which informs tutorial design for developers.

Contribution

It provides the first direct comparison of text versus video tutorials in software learning, revealing their relative effectiveness and user preferences through controlled experiments.

Findings

Text tutorials enable faster completion times.

Video tutorials facilitate quicker application of learned content.

Users prefer videos for initial learning and texts for information lookup.

Abstract

The better developers can learn software tools, the faster they can start using them and the more efficiently they can later work with them. Tutorials are supposed to help here. While in the early days of computing, mostly text tutorials were available, nowadays software developers can choose among a huge number of tutorials for almost any popular software tool. However, only little research was conducted to understand how text tutorials differ from other tutorials, which tutorial types are preferred and, especially, which tutorial types yield the best learning experience in terms of efficiency and effectiveness, especially for programmers. To evaluate these questions, we converted an existing video tutorial for a novel software tool into a content-equivalent text tutorial. We then conducted an experiment in three groups where 42 undergraduate students from a software engineering course were commissioned to operate the software tool after using a tutorial: the first group was provided only with the video tutorial, the second group only with the text tutorial and the third group with both. In this context, the differences in terms of efficiency were almost negligible: We could observe that participants using only the text tutorial completed the tutorial faster than the participants with the video tutorial. However, the participants using only the video tutorial applied the learned content faster, achieving roughly the same bottom line performance. We also found that if both tutorial types are offered, participants prefer video tutorials for learning new content but text tutorials for looking up "missed" information. We mainly gathered our data through questionnaires and screen recordings and analyzed it with suitable statistical hypotheses tests. The data is available at [12]. Since producing tutorials requires effort, knowing with which type of tutorial learnability can be increased to which extent has an immense practical relevance. We conclude that in contexts similar to ours, while it would be ideal if software tool makers would offer both tutorial types, it seems more efficient to produce only text tutorials instead of a passive video tutorial - provided you manage to motivate your learners to use them.