Dynamic Witnesses for Static Type Errors (or, Ill-Typed Programs Usually Go Wrong)

TL;DR

This paper introduces a dynamic method for explaining static type errors by generating counterexample inputs, which helps users understand and debug type errors more effectively, especially for novice programmers.

Contribution

The authors develop a symbolic execution-based technique to synthesize witnesses for ill-typed programs and extend it to produce interactive reduction graphs for debugging.

Findings

Generated witnesses for 85% of ill-typed programs

Reduction graphs provided small counterexamples in over 80% of cases

Witnesses improved students' understanding of type errors by 70%

Abstract

Static type errors are a common stumbling block for newcomers to typed functional languages. We present a dynamic approach to explaining type errors by generating counterexample witness inputs that illustrate how an ill-typed program goes wrong. First, given an ill-typed function, we symbolically execute the body to synthesize witness values that make the program go wrong. We prove that our procedure synthesizes general witnesses in that if a witness is found, then for all inhabited input types, there exist values that can make the function go wrong. Second, we show how to extend this procedure to produce a reduction graph that can be used to interactively visualize and debug witness executions. Third, we evaluate the coverage of our approach on two data sets comprising over 4,500 ill-typed student programs. Our technique is able to generate witnesses for around 85% of the programs, our reduction graph yields small counterexamples for over 80% of the witnesses, and a simple heuristic allows us to use witnesses to locate the source of type errors with around 70% accuracy. Finally, we evaluate whether our witnesses help students understand and fix type errors, and find that students presented with our witnesses show a greater understanding of type errors than those presented with a standard error message.