Automated Questions About Learners' Own Code Help to Detect Fragile Knowledge

TL;DR

This study investigates whether automatically generated questions about students' own code can reveal fragile understanding, showing that students often struggle with these questions even when their code functions correctly.

Contribution

The paper replicates and extends prior research by applying automatic question generation to a follow-up course, demonstrating its effectiveness in detecting fragile knowledge.

Findings

27% of students failed at least one generated question

Students struggling with questions had lower overall course scores

Automatic questions can reveal fragile understanding not evident from code correctness

Abstract

Students are able to produce correctly functioning program code even though they have a fragile understanding of how it actually works. Questions derived automatically from individual exercise submissions (QLC) can probe if and how well the students understand the structure and logic of the code they just created. Prior research studied this approach in the context of the first programming course. We replicate the study on a follow-up programming course for engineering students which contains a recap of general concepts in CS1. The task was the classic rainfall problem which was solved by 90% of the students. The QLCs generated from each passing submission were kept intentionally simple, yet 27% of the students failed in at least one of them. Students who struggled with questions about their own program logic had a lower median for overall course points than students who answered correctly.