Testing and Debugging Techniques for Answer Set Solver Development

TL;DR

This paper introduces automated testing and debugging methods for answer set solvers, including a grammar-based fuzz testing tool and delta debugging techniques to identify and fix solver defects.

Contribution

It presents a novel grammar-based fuzz testing tool and delta debugging algorithms tailored for answer set solver development, improving defect detection and input shrinking.

Findings

The fuzz testing tool effectively reveals unsound and incomplete behaviors in solvers.

Delta debugging algorithms successfully shrink failure-inducing inputs.

Evaluation shows improved defect detection and input reduction strategies.

Abstract

This paper develops automated testing and debugging techniques for answer set solver development. We describe a flexible grammar-based black-box ASP fuzz testing tool which is able to reveal various defects such as unsound and incomplete behavior, i.e. invalid answer sets and inability to find existing solutions, in state-of-the-art answer set solver implementations. Moreover, we develop delta debugging techniques for shrinking failure-inducing inputs on which solvers exhibit defective behavior. In particular, we develop a delta debugging algorithm in the context of answer set solving, and evaluate two different elimination strategies for the algorithm.