Describing Console I/O Behavior for Testing Student Submissions in Haskell

TL;DR

This paper introduces a formal language for specifying and testing console I/O behavior in Haskell programs, aiding in education by verifying student submissions and generating feedback.

Contribution

It presents a novel specification language with semantics based on execution traces, enabling probabilistic testing and potential educational applications.

Findings

Allows probabilistic checking of program behavior

Supports dynamic behavior with global variables

Facilitates automated feedback and exercise generation

Abstract

We present a small, formal language for specifying the behavior of simple console I/O programs. The design is driven by the concrete application case of testing interactive Haskell programs written by students. Specifications are structurally similar to lexical analysis regular expressions, but are augmented with features like global variables that track state and history of program runs, enabling expression of an interesting range of dynamic behavior. We give a semantics for our specification language based on acceptance of execution traces. From this semantics we derive a definition of the set of all traces valid for a given specification. Sampling that set enables us to mechanically check program behavior against specifications in a probabilistic fashion. Beyond testing, other possible uses of the specification language in an education context include related activities like providing more helpful feedback, generating sample solutions, and even generating random exercise tasks.