A Causal Framework for Estimating Heterogeneous Effects of On-Demand Tutoring

TL;DR

This paper presents a scalable causal inference framework to evaluate the immediate effects of on-demand human tutoring within adaptive learning systems, addressing confounding factors and heterogeneity in student responses.

Contribution

It introduces an integrated approach combining sample construction, Deep Knowledge Tracing, and Causal Forests to estimate session-level tutoring effects with robustness and granularity.

Findings

Requesting tutoring increases next-problem correctness by ~4 percentage points.

Effects vary significantly across sessions and students, from -20.25pp to +19.91pp.

Behavioral indicators like talk time do not reliably predict high-impact tutoring sessions.

Abstract

This paper introduces a scalable causal inference framework for estimating the immediate, session-level effects of on-demand human tutoring embedded within adaptive learning systems. Because students seek assistance at moments of difficulty, conventional evaluation is confounded by self-selection and time-varying knowledge states. We address these challenges by integrating principled analytic sample construction with Deep Knowledge Tracing (DKT) to estimate latent mastery, followed by doubly robust estimation using Causal Forests. Applying this framework to over 5,000 middle-school mathematics tutoring sessions, we find that requesting human tutoring increases next-problem correctness by approximately 4 percentage points and accuracy on the subsequent skill encountered by approximately 3 percentage points, suggesting that the effects of tutoring have proximal transfer across knowledge components. This effect is robust to various forms of model specification and potential unmeasured confounders. Notably, these effects exhibit significant heterogeneity across sessions and students, with session-level effect estimates ranging from $-20.25pp$ to $+19.91pp$. Our follow-up analyses suggest that typical behavioral indicators, such as student talk time, do not consistently correlate with high-impact sessions. Furthermore, treatment effects are larger for students with lower prior mastery and slightly smaller for low-SES students. This framework offers a rigorous, practical template for the evaluation and continuous improvement of on-demand human tutoring, with direct applications for emerging AI tutoring systems.