ConceptKT: A Benchmark for Concept-Level Deficiency Prediction in Knowledge Tracing

TL;DR

This paper introduces ConceptKT, a new benchmark dataset for predicting student deficiencies at the concept level in knowledge tracing, enabling more precise diagnostic feedback for personalized learning.

Contribution

It presents a novel concept-level deficiency prediction task, a dataset with concept annotations, and evaluates LLMs and LRMs for improved diagnostic accuracy in knowledge tracing.

Findings

Response history selection based on semantic similarity improves prediction accuracy.

Conceptual alignment strategies enhance deficiency identification.

Large language models show promise in fine-grained diagnostic tasks.

Abstract

Knowledge Tracing (KT) is a critical technique for modeling student knowledge to support personalized learning. However, most KT systems focus on binary correctness prediction and cannot diagnose the underlying conceptual misunderstandings that lead to errors. Such fine-grained diagnostic feedback is essential for designing targeted instruction and effective remediation. In this work, we introduce the task of concept-level deficiency prediction, which extends traditional KT by identifying the specific concepts a student is likely to struggle with on future problems. We present ConceptKT, a dataset annotated with labels that capture both the concepts required to solve each question and the missing concepts underlying incorrect responses. We investigate in-context learning approaches to KT and evaluate the diagnostic capabilities of various Large Language Models (LLMs) and Large Reasoning Models (LRMs). Different strategies for selecting informative historical records are explored. Experimental results demonstrate that selecting response histories based on conceptual alignment and semantic similarity leads to improved performance on both correctness prediction and concept-level deficiency identification.