Interpretable Difficulty-Aware Knowledge Tracing in Tutor-Student Dialogues

TL;DR

This paper introduces an interpretable, difficulty-aware knowledge tracing framework for tutor-student dialogues that explicitly models student abilities and question difficulty using LLMs and Item Response Theory.

Contribution

It presents a novel framework that combines LLMs with cognitive theories to improve interpretability and accuracy in dialogue-based knowledge tracing.

Findings

Outperforms existing KT baselines on two datasets.

Provides interpretable predictions aligned with cognitive theories.

Effectively models question difficulty and student ability explicitly.

Abstract

Recent advances in large language models (LLMs) have led to the development of AI-powered tutoring systems that provide interactive support via dialogue. To enable these tutoring systems to provide personalized support, it is essential to assess student performance at each turn, motivating knowledge tracing (KT) in dialogue settings. However, existing dialogue-based KT approaches often ignore question difficulty modeling and rely on opaque latent representations from LLMs, hindering accurate and interpretable prediction. In this work, we propose an interpretable difficulty-aware conversational KT framework built upon LLMs, which explicitly models students' abilities and the difficulty of tutor-posed tasks at each turn. The framework incorporates the original textual question and the next tutor-posed task to estimate the student's knowledge state and the difficulty of the upcoming turn. Furthermore, it integrates Item Response Theory to map LLM's outputs into student ability and question difficulty parameters, enabling interpretable prediction of student performance grounded in cognitive theories of learning. We evaluate the framework on two tutor-student dialogue datasets. Both quantitative and qualitative results show that our framework outperforms existing KT baselines, meanwhile generating interpretable outputs consistent with cognitive theory.