Design and analysis of experiments linking on-line drilling methods to improvements in knowledge

TL;DR

This study develops and analyzes an online drilling system, the tutor-web, to classify students based on their problem-solving behavior and assesses how different student types improve over a calculus course.

Contribution

The paper introduces a new student classification method using online activity data and demonstrates its potential for linking online learning behaviors to educational improvements.

Findings

Poor students show least improvement in grades.

Learners improve the most over the course.

System design can be optimized for future experiments.

Abstract

An on-line drilling system, the tutor-web, has been developed and used for teaching mathematics and statistics. The system was used in a basic course in calculus including 182 students. The students were requested to answer quiz questions in the tutor-web and therefore monitored continuously during the semester. Data available are grades on a status exam conducted in the beginning of the course, a final grade and data gathered in the tutor-web system. A classification of the students is proposed using the data gathered in the system; a Good student should be able to solve a problem quickly and get it right, the "diligent" hard-working Learner may take longer to get the right answer, a guessing (Poor) student will not take long to get the wrong answer and the remaining (Unclassified) apparent non-learning students take long to get the wrong answer, resulting in a simple classification GLUP. The (Poor) students were found to show the least improvement, defined as the change in grade from the status to the final exams, while the Learners were found to improve the most. The results are used to demonstrate how further experiments are needed and can be designed as well as to indicate how a system needs to be further developed to accommodate such experiments.