Higher Degree Inexact Model for Optimization problems

TL;DR

This paper introduces a new inexact higher degree model for optimization functions, develops adaptive gradient methods based on it, and demonstrates their effectiveness on non-smooth problems.

Contribution

It proposes a novel inexact higher degree $(elta, L, q)$-model, extending existing models, and designs universal gradient methods that work under weaker smoothness assumptions.

Findings

Convergence rates of $O(1/k + elta/k^{q/2})$ and $O(1/k^2 + elta/k^{(3q-2)/2})$ for gradient and fast gradient methods.

No error accumulation in fast gradient method for $q \u2265 2/3$, improving robustness.

Numerical experiments confirm the effectiveness of the new inexact model and methods.

Abstract

In this paper, it was proposed a new concept of the inexact higher degree $(\delta, L, q)$-model of a function that is a generalization of the inexact $(\delta, L)$-model, $(\delta, L)$-oracle and $(\delta, L)$-oracle of degree $q \in [0,2)$. Some examples were provided to illustrate the proposed new model. Adaptive inexact gradient and fast gradient methods for convex and strongly convex functions were constructed and analyzed using the new proposed inexact model. A universal fast gradient method that allows solving optimization problems with a weaker level of smoothness, among them non-smooth problems was proposed. For convex optimization problems it was proved that the proposed gradient and fast gradient methods could be converged with rates $O\left(\frac{1}{k} + \frac{\delta}{k^{q/2}}\right)$ and $O\left(\frac{1}{k^2} + \frac{\delta}{k^{(3q-2)/2}}\right)$, respectively. For the gradient method, the coefficient of $\delta$ diminishes with $k$, and for the fast gradient method, there is no error accumulation for $q \geq 2/3$. It proposed a definition of an inexact higher degree oracle for strongly convex functions and a projected gradient method using this inexact oracle. For variational inequalities and saddle point problems, a higher degree inexact model and an adaptive method called Generalized Mirror Prox to solve such class of problems using the proposed inexact model were proposed. Some numerical experiments were conducted to demonstrate the effectiveness of the proposed inexact model, we test the universal fast gradient method to solve some non-smooth problems with a geometrical nature.