KASER: Knowledge-Aligned Student Error Simulator for Open-Ended Coding Tasks

TL;DR

KASER is a novel reinforcement learning-based method that aligns student error simulation with knowledge, improving diversity and accuracy in predicting student responses for open-ended coding tasks.

Contribution

The paper introduces KASER, a new approach that enhances student error simulation by aligning errors with knowledge using a hybrid reward in reinforcement learning.

Findings

Outperforms baselines in code and error prediction at the student-problem level.

Achieves better error coverage and code diversity at the problem level.

Effectively models diverse student responses in real-world datasets.

Abstract

Open-ended tasks, such as coding problems that are common in computer science education, provide detailed insights into student knowledge. However, training large language models (LLMs) to simulate and predict possible student errors in their responses to these problems can be challenging: they often suffer from mode collapse and fail to fully capture the diversity in syntax, style, and solution approach in student responses. In this work, we present KASER (Knowledge-Aligned Student Error Simulator), a novel approach that aligns errors with student knowledge. We propose a training method based on reinforcement learning using a hybrid reward that reflects three aspects of student code prediction: i) code similarity to the ground-truth, ii) error matching, and iii) code prediction diversity. On two real-world datasets, we perform two levels of evaluation and show that: At the per-student-problem pair level, our method outperforms baselines on code and error prediction; at the per-problem level, our method outperforms baselines on error coverage and simulated code diversity.