Complexity Analysis of the Regular Simplicial Search Method with Reflection and Shrinking Steps for Derivative-Free Optimization

TL;DR

This paper introduces the regular simplicial search method (RSSM), a new derivative-free optimization algorithm with proven worst-case complexity bounds across various convexity settings, enhancing theoretical understanding of simplex-type methods.

Contribution

The paper proposes RSSM, a practical algorithm with reflection and shrinking steps, and establishes its worst-case complexity bounds for nonconvex, convex, and strongly convex problems.

Findings

Provides worst-case complexity bounds for RSSM

Guarantees convergence rates in different convexity scenarios

Lays foundation for analyzing advanced simplex-type algorithms

Abstract

Simplex-type methods, such as the well-known Nelder-Mead algorithm, are widely used in derivative-free optimization (DFO), particularly in practice. Despite their popularity, the theoretical understanding of their convergence properties has been limited, and until very recently essentially no worst-case complexity bounds were available. Recently, Cao et al. provided a sharp error bound for linear interpolation and extrapolation and derived a worst-case complexity result for a basic simplex-type method. Motivated by this, we propose a practical and provable algorithm -- the regular simplicial search method (RSSM), that incorporates reflection and shrinking steps, akin to the original method of Spendley et al. We establish worst-case complexity bounds in nonconvex, convex, and strongly convex cases. These results provide guarantees on convergence rates and lay the groundwork for future complexity analysis of more advanced simplex-type algorithms.