Understanding Student Effort Using Response-Time Propensities During Problem Solving

TL;DR

This study investigates how step-by-step response times in algebra tutoring systems can serve as a scalable measure of student effort, revealing its complex relationship with learning efficiency across different proficiency levels.

Contribution

It introduces hierarchical models to estimate response-time propensities, demonstrating their stability and conditional relation to learning outcomes in real classroom settings.

Findings

Slower response-time propensities correlate with greater learning efficiency in higher-proficiency students.

Response-time propensities are stable within students and vary with context and learner proficiency.

Early in practice, response-time data can help detect disengagement and low persistence.

Abstract

Adaptive learning systems can produce substantial learning gains, yet many students engage for too brief or too superficial a period to benefit. A central obstacle is measuring effort. Effort during multi-step problem solving is rarely directly observed, and common log-based proxies, such as time on task, cannot distinguish between a student working carefully and a student encountering a harder problem. We examine step-to-step response time as a scalable effort signal by modeling trait-like differences in students' typical response timing during tutoring (while adjusting for skill difficulty). Using step-level logs from eight classroom deployments of algebra tutoring systems (2020 to 2023) across six U.S. schools (794 students), we estimate student- and knowledge-component-level propensities using hierarchical models and relate them to learning efficiency, defined as performance improvement per completed solution step. Response-time propensities show moderate to strong stability within students, supporting their use as an individual differences measure beyond correctness. At the same time, their relationship to learning is not uniform but conditional on the learner and context. Slower propensities predict greater learning efficiency for higher-proficiency students, consistent with constructive processing, whereas for lower-proficiency students, slower propensities are weakly related or even negative, consistent with unproductive struggle or idling. These associations are strongest early in practice sequences and attenuate later in the class period, highlighting an actionable window for detecting emerging disengagement and low persistence. Overall, response-time propensities provide a practical way to incorporate temporal process data into learner models and to target adaptive supports when effort is most diagnostic.