Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs

TL;DR

This paper explores using large language models to support knowledge tracing in tutoring dialogues, demonstrating that LLM-based methods outperform existing techniques in predicting student responses and tracking knowledge over time.

Contribution

The study introduces a novel LLM-based knowledge tracing method, LLMKT, which effectively identifies skills and predicts responses in open-ended tutoring dialogues.

Findings

LLMKT outperforms existing KT methods in accuracy

LLMs can identify knowledge components in dialogue turns

Qualitative analysis reveals challenges and future directions

Abstract

Recent advances in large language models (LLMs) have led to the development of artificial intelligence (AI)-powered tutoring chatbots, showing promise in providing broad access to high-quality personalized education. Existing works have studied how to make LLMs follow tutoring principles, but have not studied broader uses of LLMs for supporting tutoring. Up until now, tracing student knowledge and analyzing misconceptions has been difficult and time-consuming to implement for open-ended dialogue tutoring. In this work, we investigate whether LLMs can be supportive of this task: we first use LLM prompting methods to identify the knowledge components/skills involved in each dialogue turn, i.e., a tutor utterance posing a task or a student utterance that responds to it. We also evaluate whether the student responds correctly to the tutor and verify the LLM's accuracy using human expert annotations. We then apply a range of knowledge tracing (KT) methods on the resulting labeled data to track student knowledge levels over an entire dialogue. We conduct experiments on two tutoring dialogue datasets, and show that a novel yet simple LLM-based method, LLMKT, significantly outperforms existing KT methods in predicting student response correctness in dialogues. We perform extensive qualitative analyses to highlight the challenges in dialogueKT and outline multiple avenues for future work.