Inline Tests

TL;DR

Inline tests are a new approach to testing individual program statements, making fault detection more precise and easier to implement, demonstrated through a framework supporting Python and Java.

Contribution

The paper introduces I-Test, the first inline testing framework, and defines language-agnostic requirements for inline testing tools.

Findings

I-Test successfully tested 144 statements in open-source projects.

Participants found inline tests easy to write and beneficial.

Inline tests helped identify two faults in real projects.

Abstract

Unit tests are widely used to check source code quality, but they can be too coarse-grained or ill-suited for testing individual program statements. We introduce inline tests to make it easier to check for faults in statements. We motivate inline tests through several language features and a common testing scenario in which inline tests could be beneficial. For example, inline tests can allow a developer to test a regular expression in place. We also define language-agnostic requirements for inline testing frameworks. Lastly, we implement I-Test, the first inline testing framework. I-Test works for Python and Java, and it satisfies most of the requirements. We evaluate I-Test on open-source projects by using it to test 144 statements in 31 Python programs and 37 Java programs. We also perform a user study. All nine user study participants say that inline tests are easy to write and that inline testing is beneficial. The cost of running inline tests is negligible, at 0.007x--0.014x, and our inline tests helped find two faults that have been fixed by the developers.