Variable Metric Random Pursuit
Sebastian U. Stich, Christian L. M\"uller, Bernd G\"artner

TL;DR
This paper introduces Variable Metric Random Pursuit, a zeroth-order optimization algorithm for smooth convex functions, with refined convergence analysis and comparisons to existing derivative-free methods.
Contribution
It presents a novel variable metric approach for Random Pursuit, including new convergence bounds and analysis of metric matching to local geometry.
Findings
V-RP converges efficiently on strongly convex functions.
V-RP outperforms some existing derivative-free algorithms in experiments.
The metric's alignment with local geometry significantly affects convergence rate.
Abstract
We consider unconstrained randomized optimization of smooth convex objective functions in the gradient-free setting. We analyze Random Pursuit (RP) algorithms with fixed (F-RP) and variable metric (V-RP). The algorithms only use zeroth-order information about the objective function and compute an approximate solution by repeated optimization over randomly chosen one-dimensional subspaces. The distribution of search directions is dictated by the chosen metric. Variable Metric RP uses novel variants of a randomized zeroth-order Hessian approximation scheme recently introduced by Leventhal and Lewis (D. Leventhal and A. S. Lewis., Optimization 60(3), 329--245, 2011). We here present (i) a refined analysis of the expected single step progress of RP algorithms and their global convergence on (strictly) convex functions and (ii) novel convergence bounds for V-RP on strongly convex…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsStochastic Gradient Optimization Techniques · Sparse and Compressive Sensing Techniques · Advanced Optimization Algorithms Research
