Exploring the Link Between Test Suite Quality and Automatic Specification Inference

TL;DR

This paper investigates how different test suite attributes affect the quality of automatically inferred specifications using invariant extraction tools, highlighting correlations with high recall in specification inference.

Contribution

It identifies key test suite attributes that influence the effectiveness of automatic specification inference, providing insights for improving inference quality.

Findings

Instruction, branch, and method coverage are correlated with high recall.

High recall values can reach up to 97.93%.

Test suite quality impacts automatic specification inference effectiveness.

Abstract

While no one doubts the importance of correct and complete specifications, many industrial systems still do not have formal specifications written out -- and even when they do, it is hard to check their correctness and completeness. This work explores the possibility of using an invariant extraction tool such as Daikon to automatically infer specifications from available test suites with the idea of aiding software engineers to improve the specifications by having another version to compare to. Given that our initial experiments did not produce satisfactory results, in this paper we explore which test suite attributes influence the quality of the inferred specification. Following further study, we found that instruction, branch and method coverage are correlated to high recall values, reaching up to 97.93%.