Arnoldi-based Sampling for High-dimensional Optimization using Imperfect Data

TL;DR

This paper introduces the Stochastic Arnoldi's Method (SAM), a high-dimensional optimization technique that uses Arnoldi-based sampling to estimate Hessian spectra from noisy data, enabling efficient optimization in complex, chaotic systems.

Contribution

The paper develops a novel Arnoldi-based sampling strategy and integrates it into a trust-region framework for high-dimensional, noisy optimization problems, demonstrating its effectiveness.

Findings

SAM outperforms traditional methods in high-dimensional noisy settings.

Arnoldi-based sampling accurately estimates Hessian eigenvalues despite data noise.

The approach is effective for optimization problems with chaotic or time-averaged objectives.

Abstract

We present a sampling strategy suitable for optimization problems characterized by high-dimensional design spaces and noisy outputs. Such outputs can arise, for example, in time-averaged objectives that depend on chaotic states. The proposed sampling method is based on a generalization of Arnoldi's method used in Krylov iterative methods. We show that Arnoldi-based sampling can effectively estimate the dominant eigenvalues of the underlying Hessian, even in the presence of inaccurate gradients. This spectral information can be used to build a low-rank approximation of the Hessian in a quadratic model of the objective. We also investigate two variants of the linear term in the quadratic model: one based on step averaging and one based on directional derivatives. The resulting quadratic models are used in a trust-region optimization framework called the Stochastic Arnoldi's Method (SAM). Numerical experiments highlight the potential of SAM relative to conventional derivative-based and derivative-free methods when the design space is high-dimensional and noisy.