Gradient Methods for Stochastic Optimization in Relative Scale

TL;DR

This paper introduces new first-order stochastic optimization methods that utilize relatively inexact subgradients, enabling efficient large-scale convex optimization with relative accuracy guarantees, exemplified by applications to eigenvector computations.

Contribution

It presents a novel concept of relatively inexact stochastic subgradients and develops first-order methods that leverage these for large-scale convex optimization.

Findings

Methods can solve large-scale SDP problems with relative accuracy.

Algorithms use only matrix-vector products for efficiency.

The approach applies to eigenvector approximation algorithms.

Abstract

We propose a new concept of a relatively inexact stochastic subgradient and present novel first-order methods that can use such objects to approximately solve convex optimization problems in relative scale. An important example where relatively inexact subgradients naturally arise is given by the Power or Lanczos algorithms for computing an approximate leading eigenvector of a symmetric positive semidefinite matrix. Using these algorithms as subroutines in our methods, we get new optimization schemes that can provably solve certain large-scale Semidefinite Programming problems with relative accuracy guarantees by using only matrix-vector products.