On the evaluation of (meta-)solver approaches

TL;DR

This paper reviews various performance metrics for evaluating meta-solver approaches, highlighting their advantages and limitations, to improve assessment methods in solver research.

Contribution

It provides a comprehensive overview of existing evaluation metrics for meta-solvers, analyzing their strengths and weaknesses based on recent studies.

Findings

Different metrics offer varying insights into meta-solver performance.

Some metrics better capture solution quality than runtime.

Evaluation methods need careful selection to accurately assess meta-solvers.

Abstract

Meta-solver approaches exploits a number of individual solvers to potentially build a better solver. To assess the performance of meta-solvers, one can simply adopt the metrics typically used for individual solvers (e.g., runtime or solution quality), or employ more specific evaluation metrics (e.g., by measuring how close the meta-solver gets to its virtual best performance). In this paper, based on some recently published works, we provide an overview of different performance metrics for evaluating (meta-)solvers, by underlying their strengths and weaknesses.