Algorithm Configuration: Learning policies for the quick termination of poor performers

TL;DR

This paper introduces a domain-adaptive method for quickly terminating poor algorithm configurations during the configuration process, using short runs to efficiently identify top performers without missing promising options.

Contribution

It proposes a novel 'performance envelope' method that learns when to terminate runs, adapting automatically to different domains to improve algorithm configuration efficiency.

Findings

Significant differences in performance linkages across domains.

The 'performance envelope' method effectively predicts when to terminate runs.

The approach accelerates configuration by reducing unnecessary long runs.

Abstract

One way to speed up the algorithm configuration task is to use short runs instead of long runs as much as possible, but without discarding the configurations that eventually do well on the long runs. We consider the problem of selecting the top performing configurations of the Conditional Markov Chain Search (CMCS), a general algorithm schema that includes, for examples, VNS. We investigate how the structure of performance on short tests links with those on long tests, showing that significant differences arise between test domains. We propose a "performance envelope" method to exploit the links; that learns when runs should be terminated, but that automatically adapts to the domain.