LLM-itation is the Sincerest Form of Data: Generating Synthetic Buggy Code Submissions for Computing Education

TL;DR

This paper demonstrates that large language models can generate synthetic buggy code submissions that closely resemble real student data, addressing privacy concerns and aiding computing education research.

Contribution

It introduces a method using GPT-4o to create realistic synthetic buggy code data, enabling privacy-preserving research and development in computing education.

Findings

Synthetic data closely matches real student test failure distributions.

LLMs can generate diverse, realistic incorrect code submissions.

Synthetic datasets can support research without compromising student privacy.

Abstract

There is a great need for data in computing education research. Data is needed to understand how students behave, to train models of student behavior to optimally support students, and to develop and validate new assessment tools and learning analytics techniques. However, relatively few computing education datasets are shared openly, often due to privacy regulations and issues in making sure the data is anonymous. Large language models (LLMs) offer a promising approach to create large-scale, privacy-preserving synthetic data, which can be used to explore various aspects of student learning, develop and test educational technologies, and support research in areas where collecting real student data may be challenging or impractical. This work explores generating synthetic buggy code submissions for introductory programming exercises using GPT-4o. We compare the distribution of test case failures between synthetic and real student data from two courses to analyze the accuracy of the synthetic data in mimicking real student data. Our findings suggest that LLMs can be used to generate synthetic incorrect submissions that are not significantly different from real student data with regard to test case failure distributions. Our research contributes to the development of reliable synthetic datasets for computing education research and teaching, potentially accelerating progress in the field while preserving student privacy.