Universal Intermediate Gradient Method for Convex Problems with Inexact Oracle

TL;DR

This paper introduces a new universal intermediate gradient method for convex optimization that adapts to unknown smoothness levels and balances convergence speed with oracle error, applicable to composite functions.

Contribution

The paper presents a novel universal intermediate gradient method that automatically adjusts to local smoothness and interpolates between existing universal gradient methods.

Findings

Method does not require prior knowledge of H"older parameters.

Achieves adaptive convergence rates based on local smoothness.

Enhanced convergence with restart technique under strong convexity.

Abstract

In this paper, we propose new first-order methods for minimization of a convex function on a simple convex set. We assume that the objective function is a composite function given as a sum of a simple convex function and a convex function with inexact H\"older-continuous subgradient. We propose Universal Intermediate Gradient Method. Our method enjoys both the universality and intermediateness properties. Following the paper by Y. Nesterov (Math.Prog., 2015) on Universal Gradient Methods, our method does not require any information about the H\"older parameter and constant and adjusts itself automatically to the local level of smoothness. On the other hand, in the spirit of the preprint by O. Devolder, F.Glineur, and Y. Nesterov (CORE DP 2013/17), our method is intermediate in the sense that it interpolates between Universal Gradient Method and Universal Fast Gradient Method. This allows to balance the rate of convergence of the method and rate of the oracle error accumulation. Under additional assumption of strong convexity of the objective, we show how the restart technique can be used to obtain an algorithm with faster rate of convergence.