AI2T: Building Trustable AI Tutors by Interactively Teaching a Self-Aware Learning Agent

TL;DR

AI2T is an interactively teachable AI system that, with minimal training, learns to accurately model problem-solving steps for intelligent tutoring systems, offering a reliable and data-efficient approach to authoring complex educational software.

Contribution

The paper introduces AI2T, a self-aware learning agent that rapidly induces robust solution-tracking rules through interactive teaching, outperforming existing methods in certainty estimation and reliability.

Findings

AI2T can be trained in 20-30 minutes to induce effective solution-tracking rules.

STAND algorithm outperforms state-of-the-art certainty estimation methods.

AI2T programs are more reliable than hallucination-prone LLMs and prior authoring approaches.

Abstract

AI2T is an interactively teachable AI for authoring intelligent tutoring systems (ITSs). Authors tutor AI2T by providing a few step-by-step solutions and then grading AI2T's own problem-solving attempts. From just 20-30 minutes of interactive training, AI2T can induce robust rules for step-by-step solution tracking (i.e., model-tracing). As AI2T learns it can accurately estimate its certainty of performing correctly on unseen problem steps using STAND: a self-aware precondition learning algorithm that outperforms state-of-the-art methods like XGBoost. Our user study shows that authors can use STAND's certainty heuristic to estimate when AI2T has been trained on enough diverse problems to induce correct and complete model-tracing programs. AI2T-induced programs are more reliable than hallucination-prone LLMs and prior authoring-by-tutoring approaches. With its self-aware induction of hierarchical rules, AI2T offers a path toward trustable data-efficient authoring-by-tutoring for complex ITSs that normally require as many as 200-300 hours of programming per hour of instruction.