Automated Knowledge Component Generation for Interpretable Knowledge Tracing in Coding Problems

TL;DR

This paper introduces an automated pipeline using large language models to generate and tag knowledge components for programming problems, improving interpretability and prediction accuracy in knowledge tracing.

Contribution

It presents a novel LLM-based method for automatic KC generation and tagging, reducing manual effort and enhancing student response prediction in coding education.

Findings

KCGen-KT outperforms existing methods and human KCs in response prediction.

Generated KCs better fit cognitive models than human-written KCs.

Human evaluation confirms the accuracy of problem-KC mappings.

Abstract

Knowledge components (KCs) mapped to problems help model student learning, tracking their mastery levels on fine-grained skills thereby facilitating personalized learning and feedback in online learning platforms. However, crafting and tagging KCs to problems, traditionally performed by human domain experts, is highly labor intensive. We present an automated, LLM-based pipeline for KC generation and tagging for open-ended programming problems. We also develop an LLM-based knowledge tracing (KT) framework to leverage these LLM-generated KCs, which we refer to as KCGen-KT. We conduct extensive quantitative and qualitative evaluations on two real-world student code submission datasets in different programming languages.We find that KCGen-KT outperforms existing KT methods and human-written KCs on future student response prediction. We investigate the learning curves of generated KCs and show that LLM-generated KCs result in a better fit than human written KCs under a cognitive model. We also conduct a human evaluation with course instructors to show that our pipeline generates reasonably accurate problem-KC mappings.