Leveraging Generative AI for Enhancing Automated Assessment in Programming Education Contests

TL;DR

This paper presents a novel NLP-based approach using generative AI to automate high-quality test case creation for programming contests, significantly improving assessment effectiveness and reducing educator workload.

Contribution

It introduces an innovative AI-driven method for automated test case generation in programming education, validated on diverse datasets and openly shared for practical use.

Findings

AI-generated test cases identified errors in 67% of problems

Enhanced assessment quality and educator efficiency

Validated across multiple international datasets

Abstract

Competitive programming contests play a crucial role in cultivating computational thinking and algorithmic skills among learners. However, generating comprehensive test cases to effectively assess programming solutions remains resource-intensive and challenging for educators. This paper introduces an innovative NLP-driven method leveraging generative AI (large language models) to automate the creation of high-quality test cases for competitive programming assessments. We extensively evaluated our approach on diverse datasets, including 25 years of Romanian Informatics Olympiad (OJI) data for 5th graders, recent competitions hosted on the Kilonova.ro platform, and the International Informatics Olympiad in Teams (IIOT). Our results demonstrate that AI-generated test cases substantially enhanced assessments, notably identifying previously undetected errors in 67% of the OJI 5th grade programming problems. These improvements underscore the complementary educational value of our technique in formative assessment contexts. By openly sharing our prompts, translated datasets, and methodologies, we offer practical NLP-based tools that educators and contest organizers can readily integrate to enhance assessment quality, reduce workload, and deepen insights into learner performance.