A promise checked is a promise kept: Inspection Testing

TL;DR

This paper introduces inspection testing, a new paradigm allowing developers to declaratively specify and automatically verify non-functional properties of compilation artifacts, improving the reliability of complex Haskell libraries.

Contribution

It formalizes inspection testing, implements it for Haskell, and demonstrates its effectiveness in enhancing library quality and ensuring compiler optimizations meet expectations.

Findings

Automates verification of compilation properties

Reduces manual inspection effort

Improves library reliability and correctness

Abstract

Occasionally, developers need to ensure that the compiler treats their code in a specific way that is only visible by inspecting intermediate or final compilation artifacts. This is particularly common with carefully crafted compositional libraries, where certain usage patterns are expected to trigger an intricate sequence of compiler optimizations -- stream fusion is a well-known example. The developer of such a library has to manually inspect build artifacts and check for the expected properties. Because this is too tedious to do often, it will likely go unnoticed if the property is broken by a change to the library code, its dependencies or the compiler. The lack of automation has led to released versions of such libraries breaking their documented promises. This indicates that there is an unrecognized need for a new testing paradigm, inspection testing, where the programmer declaratively describes non-functional properties of an compilation artifact and the compiler checks these properties. We define inspection testing abstractly, implement it in the context of Haskell and show that it increases the quality of such libraries.