Evaluation of Systems Programming Exercises through Tailored Static Analysis

TL;DR

This paper explores the use of tailored static analysis techniques to automatically evaluate systems programming exercises, providing detailed feedback and identifying errors missed by traditional test cases.

Contribution

It introduces a method for designing assignment-specific static analysis rules to improve automatic assessment accuracy in systems programming courses.

Findings

Static analysis identified many errors missed by test cases.

Tailored static analysis provides detailed feedback for students.

The approach improves evaluation efficiency in large classes.

Abstract

In large programming classes, it takes a significant effort from teachers to evaluate exercises and provide detailed feedback. In systems programming, test cases are not sufficient to assess exercises, since concurrency and resource management bugs are difficult to reproduce. This paper presents an experience report on static analysis for the automatic evaluation of systems programming exercises. We design systems programming assignments with static analysis rules that are tailored for each assignment, to provide detailed and accurate feedback. Our evaluation shows that static analysis can identify a significant number of erroneous submissions missed by test cases.