Computer Aided Design and Grading for an Electronic Functional Programming Exam

TL;DR

This paper presents new algorithms and tools to automate and improve the fairness of grading in electronic functional programming exams, demonstrating successful implementation and positive student and examiner feedback.

Contribution

Introduces a novel proof puzzle checking algorithm, a static analysis tool for source code, and a high-level language for regular expressions to enhance e-exam design and grading.

Findings

Almost all tasks can be graded automatically or semi-automatically.

Students find e-exams suitable but are split on expressing thoughts.

Examiners experience more time-efficient grading with similar score distributions.

Abstract

Electronic exams (e-exams) have the potential to substantially reduce the effort required for conducting an exam through automation. Yet, care must be taken to sacrifice neither task complexity nor constructive alignment nor grading fairness in favor of automation. To advance automation in the design and fair grading of (functional programming) e-exams, we introduce the following: A novel algorithm to check Proof Puzzles based on finding correct sequences of proof lines that improves fairness compared to an existing, edit distance based algorithm; an open-source static analysis tool to check source code for task relevant features by traversing the abstract syntax tree; a higher-level language and open-source tool to specify regular expressions that makes creating complex regular expressions less error-prone. Our findings are embedded in a complete experience report on transforming a paper exam to an e-exam. We evaluated the resulting e-exam by analyzing the degree of automation in the grading process, asking students for their opinion, and critically reviewing our own experiences. Almost all tasks can be graded automatically at least in part (correct solutions can almost always be detected as such), the students agree that an e-exam is a fitting examination format for the course but are split on how well they can express their thoughts compared to a paper exam, and examiners enjoy a more time-efficient grading process while the point distribution in the exam results was almost exactly the same compared to a paper exam.