Measuring the Effectiveness of Learning Resources Via Student Interaction with Online Learning Modules

TL;DR

This paper introduces a novel method for evaluating online learning modules by analyzing student interaction logs, providing detailed insights into module effectiveness and areas for improvement.

Contribution

The study presents a new approach combining mastery assessment, effort analysis, and visual data representation to measure online learning resource effectiveness.

Findings

Effective modules identified through log data analysis

Visual sunburst charts facilitate quick interpretation of results

Method enables rapid assessment of online learning tools

Abstract

We present a new method for measuring the effectiveness of online learning resources, through the analysis of time-stamped log data of students' interaction with a sequence of online learning modules created based on the concept of mastery learning. Each module was designed to assess students' mastery of one topic before and after interacting with the learning resources in the module. In addition, analysis of log data provides information on students' test-taking effort when completing the assessment, and learning effort when interacting with the learning resources. Combining these three measurements provides accurate information on the quality of each online learning module, as well as detailed suggestions for future improvements. The results from data collected from all 10 modules are presented in a sequence of sunburst charts, an intuitive visual representation designed to allow the average instructor to quickly grasp the key outcomes of the data analysis, and identify less effective modules for future improvements. Online learning modules can be implemented much more frequently than either clinical experiments or classroom assessments, and can provide more interpretable data than most existing online courses, making it a valuable tool for quickly assessing the effectiveness of online learning resources.