Randomized Subspace Derivative-Free Optimization with Quadratic Models and Second-Order Convergence

TL;DR

This paper introduces a new derivative-free optimization method that uses random subspaces and quadratic models, achieving second-order convergence with improved dimension dependence, enabling larger problem solving.

Contribution

It provides the first worst-case complexity bounds for subspace DFO methods with second-order convergence and introduces a practical quadratic interpolation approach for large-scale problems.

Findings

First worst-case complexity bounds for subspace DFO with second-order convergence

Significantly improved dimension dependence over full-space methods

Able to solve larger problems efficiently while maintaining performance on medium-scale problems

Abstract

We consider model-based derivative-free optimization (DFO) for large-scale problems, based on iterative minimization in random subspaces. We provide the first worst-case complexity bound for such methods for convergence to approximate second-order critical points, and show that these bounds have significantly improved dimension dependence compared to standard full-space methods, provided low accuracy solutions are desired and/or the problem has low effective rank. We also introduce a practical subspace model-based method suitable for general objective minimization, based on iterative quadratic interpolation in subspaces, and show that it can solve significantly larger problems than state-of-the-art full-space methods, while also having comparable performance on medium-scale problems when allowed to use full-dimension subspaces.