Extended Paper: API-driven Program Synthesis for Testing Static Typing Implementations

TL;DR

This paper presents Thalia, an API-driven program synthesis approach for testing static typing implementations, generating small, feature-rich test programs that uncover numerous typing bugs in popular language compilers.

Contribution

It introduces a novel method for synthesizing small, API-based test programs to effectively test static type checkers, including fault-injection and semantics-preserving modes.

Findings

Uncovered 84 typing bugs across three language compilers.

Most bugs triggered by APIs with parametric polymorphism, overloading, and higher-order functions.

Thalia produces test programs with unique characteristics compared to existing tools.

Abstract

We introduce a novel approach for testing static typing implementations based on the concept of API-driven program synthesis. The idea is to synthesize type-intensive but small and well-typed programs by leveraging and combining application programming interfaces (APIs) derived from existing software libraries. Our primary insight is backed up by real-world evidence: a significant number of compiler typing bugs are caused by small test cases that employ APIs from the standard library of the language under test. This is attributed to the inherent complexity of the majority of these APIs, which often exercise a wide range of sophisticated type-related features. The main contribution of our approach is the ability to produce small client programs with increased feature coverage, without bearing the burden of generating the corresponding well-formed API definitions from scratch. To validate diverse aspects of static typing procedures (i.e., soundness, precision of type inference), we also enrich our API-driven approach with fault-injection and semantics-preserving modes, along with their corresponding test oracles. We evaluate our implemented tool, Thalia on testing the static typing implementations of the compilers for three popular languages, namely, Scala, Kotlin, and Groovy. Thalia has uncovered 84 typing bugs (77 confirmed and 22 fixed), most of which are triggered by test cases featuring APIs that rely on parametric polymorphism, overloading, and higher-order functions. Our comparison with state-of-the-art shows that Thalia yields test programs with distinct characteristics, offering additional and complementary benefits.