Embracing Imperfection: Simulating Students with Diverse Cognitive Levels Using LLM-based Agents

TL;DR

This paper introduces a training-free framework for simulating students with diverse cognitive abilities using LLMs, by constructing cognitive prototypes and refining student solutions to better mimic realistic learning behaviors.

Contribution

It presents a novel, training-free approach that uses knowledge graphs and beam search to simulate diverse student behaviors more accurately than existing methods.

Findings

Achieved 100% improvement in simulation accuracy over baselines.

Constructed the Student_100 dataset with 5,000 learning records.

Effectively modeled diverse cognitive levels in student simulations.

Abstract

Large language models (LLMs) are revolutionizing education, with LLM-based agents playing a key role in simulating student behavior. A major challenge in student simulation is modeling the diverse learning patterns of students at various cognitive levels. However, current LLMs, typically trained as ``helpful assistants'', target at generating perfect responses. As a result, they struggle to simulate students with diverse cognitive abilities, as they often produce overly advanced answers, missing the natural imperfections that characterize student learning and resulting in unrealistic simulations. To address this issue, we propose a training-free framework for student simulation. We begin by constructing a cognitive prototype for each student using a knowledge graph, which captures their understanding of concepts from past learning records. This prototype is then mapped to new tasks to predict student performance. Next, we simulate student solutions based on these predictions and iteratively refine them using a beam search method to better replicate realistic mistakes. To validate our approach, we construct the \texttt{Student\_100} dataset, consisting of $100$ students working on Python programming and $5,000$ learning records. Experimental results show that our method consistently outperforms baseline models, achieving $100\%$ improvement in simulation accuracy.