Next Token Knowledge Tracing: Exploiting Pretrained LLM Representations to Decode Student Behaviour

TL;DR

This paper introduces Next Token Knowledge Tracing (NTKT), a novel method that uses pretrained Large Language Models to incorporate question content and student responses for improved knowledge prediction in educational settings.

Contribution

NTKT redefines knowledge tracing as a next-token prediction task with LLMs, effectively integrating question text and student history for enhanced predictive accuracy.

Findings

Significantly outperforms existing neural KT models.

Better generalization to cold-start questions and users.

Highlights importance of question content in student modeling.

Abstract

Modelling student knowledge is a key challenge when leveraging AI in education, with major implications for personalised learning. The Knowledge Tracing (KT) task aims to predict how students will respond to educational questions in learning environments, based on their prior interactions. Existing KT models typically use response correctness along with metadata like skill tags and timestamps, often overlooking the question text, which is an important source of pedagogical insight. This omission poses a lost opportunity while limiting predictive performance. We propose Next Token Knowledge Tracing (NTKT), a novel approach that reframes KT as a next-token prediction task using pretrained Large Language Models (LLMs). NTKT represents both student histories and question content as sequences of text, allowing LLMs to learn patterns in both behaviour and language. Our series of experiments significantly improves performance over state-of-the-art neural KT models and generalises much better to cold-start questions and users. These findings highlight the importance of question content in KT and demonstrate the benefits of leveraging pretrained representations of LLMs to model student learning more effectively.