Leveraging Large Language Models for Identifying Knowledge Components

TL;DR

This paper explores using large language models to automate the identification of knowledge components in educational content, addressing redundancy and performance issues through semantic merging techniques.

Contribution

It introduces a novel semantic merging method to improve LLM-generated knowledge components, enhancing accuracy and reducing redundancy.

Findings

Scaling LLMs alone yields subpar performance compared to expert models.

Semantic merging significantly reduces redundant KCs.

Optimized thresholding improves RMSE and reduces KC count.

Abstract

Knowledge Components (KCs) are foundational to adaptive learning systems, but their manual identification by domain experts is a significant bottleneck. While Large Language Models (LLMs) offer a promising avenue for automating this process, prior research has been limited to small datasets and has been shown to produce superfluous, redundant KC labels. This study addresses these limitations by first scaling a "simulated textbook" LLM prompting strategy (using GPT-4o-mini) to a larger dataset of 646 multiple-choice questions. We found that this initial automated approach performed significantly worse than an expert-designed KC model (RMSE 0.4285 vs. 0.4206) and generated an excessive number of KCs (569 vs. 101). To address the issue of redundancy, we proposed and evaluated a novel method for merging semantically similar KC labels based on their cosine similarity. This merging strategy significantly improved the model's performance; a model using a cosine similarity threshold of 0.8 achieved the best result, reducing the KC count to 428 and improving the RMSE to 0.4259. This demonstrates that while scaled LLM generation alone is insufficient, combining it with a semantic merging technique offers a viable path toward automating and refining KC identification.