Contraction methods for continuous optimization

TL;DR

This paper introduces contraction methods for continuous optimization, classifies problems based on their contractibility, and proposes an algorithm with convergence guarantees, enhancing understanding of problem difficulty and solution strategies.

Contribution

It develops a framework for contraction methods, categorizes optimization problems by contractibility, and presents a practical algorithm with convergence bounds.

Findings

Problems are classified into three categories based on contractibility.

Linear convergence is achievable for contractible problems.

The proposed algorithm demonstrates high probability bounds for convergence.

Abstract

Motivated by the grid search method and Bayesian optimization, we introduce the concept of contractibility and its applications in model-based optimization. First, a basic framework of contraction methods is established to construct a nonempty closed set sequence that contracts from the initial domain to the set of global minimizers. Then, from the perspective of whether the contraction can be carried out effectively, relevant conditions are introduced to divide all continuous optimization problems into three categories: (i) logarithmic time contractible, (ii) polynomial time contractible, or (iii) noncontractible. For every problem from the first two categories, there exists a contraction sequence that converges to the set of all global minimizers with linear convergence; for any problem from the last category, we discuss possible troubles caused by contraction. Finally, a practical algorithm is proposed with high probability bounds for convergence rate and complexity. It is shown that the contractibility contributes to practical applications and can also be seen as a complement to smoothness for distinguishing the optimization problems that are easy to solve.