Scalable Derivative-Free Optimization Algorithms with Low-Dimensional Subspace Techniques

TL;DR

This paper presents a scalable derivative-free optimization framework that leverages low-dimensional subspace techniques based on approximate gradients, demonstrating effectiveness on high-dimensional problems with only function evaluations.

Contribution

It re-introduces a subspace optimization framework with convergence analysis and shows its applicability to large-scale problems using inaccurate function data.

Findings

Effective for problems with dimensions up to 10^4

Converges globally with worst-case complexity guarantees

Works with only approximate function evaluations

Abstract

We re-introduce a derivative-free subspace optimization framework originating from Chapter 5 of the Ph.D. thesis [Z. Zhang, On Derivative-Free Optimization Methods, Ph.D. thesis, Chinese Academy of Sciences, Beijing, 2012] of the author under the supervision of Ya-xiang Yuan. At each iteration, the framework defines a (low-dimensional) subspace based on an approximate gradient, and then solves a subproblem in this subspace to generate a new iterate. We sketch the global convergence and worst-case complexity analysis of the framework, elaborate on its implementation, and present some numerical results on solving problems with dimensions as high as 10^4 using only inaccurate function values.