Automatic Assessment of the Design Quality of Python Programs with Personalized Feedback

TL;DR

This paper introduces a neural network model that automatically assesses the design quality of Python programs and provides personalized feedback, significantly aiding large-scale programming education.

Contribution

The paper presents a novel neural network approach for automatic design quality assessment and personalized feedback generation in programming education.

Findings

Model achieves 83.67% to 94.27% accuracy in predicting design scores.

Students improved their design scores by 19.58% after receiving personalized feedback.

The approach scales assessment and feedback in large classes.

Abstract

The assessment of program functionality can generally be accomplished with straight-forward unit tests. However, assessing the design quality of a program is a much more difficult and nuanced problem. Design quality is an important consideration since it affects the readability and maintainability of programs. Assessing design quality and giving personalized feedback is very time consuming task for instructors and teaching assistants. This limits the scale of giving personalized feedback to small class settings. Further, design quality is nuanced and is difficult to concisely express as a set of rules. For these reasons, we propose a neural network model to both automatically assess the design of a program and provide personalized feedback to guide students on how to make corrections. The model's effectiveness is evaluated on a corpus of student programs written in Python. The model has an accuracy rate from 83.67% to 94.27%, depending on the dataset, when predicting design scores as compared to historical instructor assessment. Finally, we present a study where students tried to improve the design of their programs based on the personalized feedback produced by the model. Students who participated in the study improved their program design scores by 19.58%.