The Configurable SAT Solver Challenge (CSSC)

TL;DR

The CSSC evaluates SAT solvers based on their performance after automated parameter tuning, highlighting the importance of configurability for solving specific problem instances effectively.

Contribution

This paper introduces the CSSC, a new competition framework that assesses solvers by their tuned performance rather than default settings.

Findings

Configurable solvers outperform default settings in tailored scenarios.

Automated configuration significantly improves solver performance.

CSSC results demonstrate the benefits of parameter tuning for SAT solving.

Abstract

It is well known that different solution strategies work well for different types of instances of hard combinatorial problems. As a consequence, most solvers for the propositional satisfiability problem (SAT) expose parameters that allow them to be customized to a particular family of instances. In the international SAT competition series, these parameters are ignored: solvers are run using a single default parameter setting (supplied by the authors) for all benchmark instances in a given track. While this competition format rewards solvers with robust default settings, it does not reflect the situation faced by a practitioner who only cares about performance on one particular application and can invest some time into tuning solver parameters for this application. The new Configurable SAT Solver Competition (CSSC) compares solvers in this latter setting, scoring each solver by the performance it achieved after a fully automated configuration step. This article describes the CSSC in more detail, and reports the results obtained in its two instantiations so far, CSSC 2013 and 2014.