Trace Typing: An Approach for Evaluating Retrofitted Type Systems (Extended Version)

TL;DR

This paper introduces trace typing, a framework for automatically evaluating and comparing different retrofitted type systems on large codebases, aiding the design of safer and more effective type systems for dynamic languages.

Contribution

The paper presents trace typing, a novel quantitative evaluation method for retrofitted type systems, enabling systematic comparison and guiding design choices on large codebases.

Findings

Compared variations of type systems on JavaScript code

Validated and guided design of a new type system for JavaScript

Automatically identified dynamic type refinement checks

Abstract

Recent years have seen growing interest in the retrofitting of type systems onto dynamically-typed programming languages, in order to improve type safety, programmer productivity, or performance. In such cases, type system developers must strike a delicate balance between disallowing certain coding patterns to keep the type system simple, or including them at the expense of additional complexity and effort. Thus far, the process for designing retrofitted type systems has been largely ad hoc, because evaluating multiple variations of a type system on large bodies of existing code is a significant undertaking. We present trace typing: a framework for automatically and quantitatively evaluating variations of a retrofitted type system on large code bases. The trace typing approach involves gathering traces of program executions, inferring types for instances of variables and expressions occurring in a trace, and merging types according to merge strategies that reflect specific (combinations of) choices in the source-level type system design space. We evaluated trace typing through several experiments. We compared several variations of type systems retrofitted onto JavaScript, measuring the number of program locations with type errors in each case on a suite of over fifty thousand lines of JavaScript code. We also used trace typing to validate and guide the design of a new retrofitted type system that enforces fixed object layout for JavaScript objects. Finally, we leveraged the types computed by trace typing to automatically identify tag tests --- dynamic checks that refine a type --- and examined the variety of tests identified.