Optimal Restart Strategies for Parameter-dependent Optimization Algorithms

TL;DR

This paper analyzes restart strategies for parameter-dependent algorithms, establishing bounds on their efficiency and identifying an optimal multiplicative increase factor that minimizes worst-case computational waste.

Contribution

It classifies restart strategies, introduces a loss function to measure efficiency, and finds an optimal multiplicative factor for strategy improvement that is independent of the unknown parameter.

Findings

Bounded loss achieved by multiplicative increase strategies

Existence of an optimal multiplicative factor for lambda

Optimal factor is independent of the unknown optimal lambda

Abstract

This paper examines restart strategies for algorithms whose successful termination depends on an unknown parameter $\lambda$. After each restart, $\lambda$ is increased, until the algorithm terminates successfully. It is assumed that there is a unique, unknown, optimal value for $\lambda$. For the algorithm to run successfully, this value must be reached or surpassed. The key question is whether there exists an optimal strategy for selecting $\lambda$ after each restart taking into account that the computational costs (runtime) increases with $\lambda$. In this work, potential restart strategies are classified into parameter-dependent strategy types. A loss function is introduced to quantify the wasted computational cost relative to the optimal strategy. A crucial requirement for any efficient restart strategy is that its loss, relative to the optimal $\lambda$, remains bounded. To this end, upper and lower bounds of the loss are derived. Using these bounds it will be shown that not all strategy types are bounded. However, for a particular strategy type, where $\lambda$ is increased multiplicatively by a constant factor $\lambda$, the relative loss function is bounded. Furthermore, it will be demonstrated that within this strategy type, there exists an optimal value for $\lambda$ that minimizes the maximum relative loss. In the asymptotic limit, this optimal choice of $\lambda$ does not depend on the unknown optimal $\lambda$.