Utilitarian Algorithm Configuration

TL;DR

This paper introduces a new approach for configuring heuristic algorithms based on a utilitarian objective, which better captures user preferences and offers theoretical guarantees, contrasting with traditional mean runtime minimization.

Contribution

The paper proposes a novel configuration procedure optimizing a utilitarian objective, providing theoretical bounds and empirical performance analysis.

Findings

Utility-based configuration offers meaningful performance bounds.

Procedures outperform mean runtime minimization in experiments.

Theoretical guarantees match lower bounds.

Abstract

We present the first nontrivial procedure for configuring heuristic algorithms to maximize the utility provided to their end users while also offering theoretical guarantees about performance. Existing procedures seek configurations that minimize expected runtime. However, very recent theoretical work argues that expected runtime minimization fails to capture algorithm designers' preferences. Here we show that the utilitarian objective also confers significant algorithmic benefits. Intuitively, this is because mean runtime is dominated by extremely long runs even when they are incredibly rare; indeed, even when an algorithm never gives rise to such long runs, configuration procedures that provably minimize mean runtime must perform a huge number of experiments to demonstrate this fact. In contrast, utility is bounded and monotonically decreasing in runtime, allowing for meaningful empirical bounds on a configuration's performance. This paper builds on this idea to describe effective and theoretically sound configuration procedures. We prove upper bounds on the runtime of these procedures that are similar to theoretical lower bounds, while also demonstrating their performance empirically.