Which Type of Students can LLMs Act? Investigating Authentic Simulation with Graph-based Human-AI Collaborative System

TL;DR

This paper introduces a graph-based collaborative system that uses LLMs and human input to generate realistic student simulations, aiding educational research and assessment.

Contribution

It presents a novel three-stage pipeline combining automated scoring, expert calibration, and graph propagation to improve student simulation fidelity.

Findings

Simulated students closely match human judgments

Graph-based propagation enhances score consistency

Profiles and behaviors are faithfully represented

Abstract

While rapid advances in large language models (LLMs) are reshaping data-driven intelligent education, accurately simulating students remains an important but challenging bottleneck for scalable educational data collection, evaluation, and intervention design. However, current works are limited by scarce real interaction data, costly expert evaluation for realism, and a lack of large-scale, systematic analyses of LLMs ability in simulating students. We address this gap by presenting a three-stage LLM-human collaborative pipeline to automatically generate and filter high-quality student agents. We leverage a two-round automated scoring validated by human experts and deploy a score propagation module to obtain more consistent scores across the student similarity graph. Experiments show that combining automated scoring, expert calibration, and graph-based propagation yields simulated student that more closely track authentication by human judgments. We then analyze which profiles and behaviors are simulated more faithfully, supporting subsequent studies on personalized learning and educational assessment.