Collaborative Management of Benchmark Instances and their Attributes

TL;DR

This paper introduces a tool for managing and retrieving SAT benchmark instances and their attributes, enhancing experimental evaluation, analysis, and sharing in algorithm research.

Contribution

It presents a novel specification of a benchmark instance identifier that simplifies managing and relating instances and attributes.

Findings

Improved management of benchmark instances and attributes.

Facilitated analysis and comparison of SAT experiments.

Enhanced sharing and reproducibility of experimental results.

Abstract

Experimental evaluation is an integral part in the design process of algorithms. Publicly available benchmark instances are widely used to evaluate methods in SAT solving. For the interpretation of results and the design of algorithm portfolios their attributes are crucial. Capturing the interrelation of benchmark instances and their attributes is considerably simplified through our specification of a benchmark instance identifier. Thus, our tool increases the availability of both by providing means to manage and retrieve benchmark instances by their attributes and vice versa. Like this, it facilitates the design and analysis of SAT experiments and the exchange of results.