Accuracy and Relationships of Quadratic Models in Derivative-free Optimization

TL;DR

This paper systematically analyzes the approximation accuracy and relationships of three quadratic models used in derivative-free optimization, providing new theoretical bounds and insights.

Contribution

It establishes fully linear error bounds for MN, MFN, and QS models without previous Hessian bounds, and explores their structural relationships.

Findings

All three models achieve fully quadratic accuracy along sample directions.

The paper removes the need for bounded Hessian assumptions in error analysis.

It characterizes conditions under which the models coincide.

Abstract

We study three quadratic models in model-based derivative-free optimization: the minimum norm (MN), minimum Frobenius norm (MFN), and quadratic generalized simplex derivative (QS) models. Despite their widespread use, their approximation accuracy and relationships have not been systematically explored. We establish fully linear error bounds for all three models, removing the uniformly bounded model Hessian assumption required in existing MN analyses and deriving the first such results for the QS model. We further analyze Hessian approximation accuracy via directional error bounds, showing that all three models achieve fully quadratic accuracy along sample directions under a mild condition on the sample set. This reveals a form of directional fully quadratic accuracy not captured by existing theory. Finally, we characterize the relationships among these models, identifying conditions under which they coincide and clarifying their structural connections.