Simulated Human Learning in a Dynamic, Partially-Observed, Time-Series Environment

TL;DR

This paper develops a simulated classroom environment with student-teacher interactions to evaluate reinforcement learning-based intelligent tutoring systems, highlighting the importance of probing interventions for improved student modeling and adaptive instruction.

Contribution

It introduces a dynamic, partially-observed time-series environment for testing ITSs and compares RL algorithms with heuristics, emphasizing the role of probing interventions in student estimation.

Findings

Probing interventions improve student state estimation.

RL and heuristic policies perform similarly overall.

Probing benefits diminish with more hidden information.

Abstract

While intelligent tutoring systems (ITSs) can use information from past students to personalize instruction, each new student is unique. Moreover, the education problem is inherently difficult because the learning process is only partially observable. We therefore develop a dynamic, time-series environment to simulate a classroom setting, with student-teacher interventions - including tutoring sessions, lectures, and exams. In particular, we design the simulated environment to allow for varying levels of probing interventions that can gather more information. Then, we develop reinforcement learning ITSs that combine learning the individual state of students while pulling from population information through the use of probing interventions. These interventions can reduce the difficulty of student estimation, but also introduce a cost-benefit decision to find a balance between probing enough to get accurate estimates and probing so often that it becomes disruptive to the student. We compare the efficacy of standard RL algorithms with several greedy rules-based heuristic approaches to find that they provide different solutions, but with similar results. We also highlight the difficulty of the problem with increasing levels of hidden information, and the boost that we get if we allow for probing interventions. We show the flexibility of both heuristic and RL policies with regards to changing student population distributions, finding that both are flexible, but RL policies struggle to help harder classes. Finally, we test different course structures with non-probing policies and we find that our policies are able to boost the performance of quiz and midterm structures more than we can in a finals-only structure, highlighting the benefit of having additional information.