Automated, Targeted Testing of Property-Based Testing Predicates
Tim Nelson (Brown University, USA), Elijah Rivera (Brown University,, USA), Sam Soucie (Indiana University, USA), Thomas Del Vecchio (Brown, University, USA), John Wrenn (Brown University, USA), Shriram Krishnamurthi, (Brown University, USA)

TL;DR
This paper introduces automated methods based on property-based testing and SAT-solving to evaluate student-written predicates, revealing limitations of previous approaches and improving assessment accuracy in PBT education.
Contribution
It challenges the independence assumption of subproperties in PBT, proposes automation techniques for predicate testing, and provides a comprehensive analysis of student performance.
Findings
Automation can outperform manual testing in predicate assessment
Relaxing independence assumptions reveals more errors and misconceptions
Proposed methods improve the evaluation of student PBT predicates
Abstract
Context: This work is based on property-based testing (PBT). PBT is an increasingly important form of software testing. Furthermore, it serves as a concrete gateway into the abstract area of formal methods. Specifically, we focus on students learning PBT methods. Inquiry: How well do students do at PBT? Our goal is to assess the quality of the predicates they write as part of PBT. Prior work introduced the idea of decomposing the predicate's property into a conjunction of independent subproperties. Testing the predicate against each subproperty gives a "semantic" understanding of their performance. Approach: The notion of independence of subproperties both seems intuitive and was an important condition in prior work. First, we show that this condition is overly restrictive and might hide valuable information: it both undercounts errors and makes it hard to capture misconceptions.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Testing and Debugging Techniques · Software Reliability and Analysis Research · Software Engineering Research
