On Testing Constraint Programs

TL;DR

This paper presents a framework and tool for automated testing of constraint programs, focusing on detecting faults introduced during model refinement using test case generation from an initial declarative model.

Contribution

It introduces a novel testing framework and tool, CPTEST, for automatically detecting non-conformities in constraint program refinements, enhancing software engineering practices.

Findings

Successfully detected non-conformities in benchmark programs

Automated test data generation from declarative models

Applicable to classical constraint problems like Golomb rulers

Abstract

The success of several constraint-based modeling languages such as OPL, ZINC, or COMET, appeals for better software engineering practices, particularly in the testing phase. This paper introduces a testing framework enabling automated test case generation for constraint programming. We propose a general framework of constraint program development which supposes that a first declarative and simple constraint model is available from the problem specifications analysis. Then, this model is refined using classical techniques such as constraint reformulation, surrogate and global constraint addition, or symmetry-breaking to form an improved constraint model that must be thoroughly tested before being used to address real-sized problems. We think that most of the faults are introduced in this refinement step and propose a process which takes the first declarative model as an oracle for detecting non-conformities. We derive practical test purposes from this process to generate automatically test data that exhibit non-conformities. We implemented this approach in a new tool called CPTEST that was used to automatically detect non-conformities on two classical benchmark programs, namely the Golomb rulers and the car-sequencing problem.