Exact Convergence rate of the subgradient method by using Polyak step size

TL;DR

This paper analyzes the convergence rates of the subgradient method with Polyak step size, introducing adaptive variants that achieve optimal rates, and also investigates the convergence of the alternating projection method.

Contribution

It establishes the exact convergence rates of the subgradient method with Polyak step size and proposes adaptive methods that attain optimal convergence rates.

Findings

Subgradient method with Polyak step size has a convergence rate of O(1/√[4]{N}) for the last iterate.

Adaptive Polyak step size improves the rate to O(1/√N), matching the lower bound.

An adaptive Polyak method with momentum also achieves the optimal convergence rate.

Abstract

This paper studies the last iterate of subgradient method with Polyak step size when applied to the minimization of a nonsmooth convex function with bounded subgradients. We show that the subgradient method with Polyak step size achieves a convergence rate $\mathcal{O}\left(\tfrac{1}{\sqrt[4]{N}}\right)$ in terms of the final iterate. An example is provided to show that this rate is exact and cannot be improved. We introduce an adaptive Polyak step size for which the subgradient method enjoys a convergence rate $\mathcal{O}\left(\tfrac{1}{\sqrt{N}}\right)$ for the last iterate. Its convergence rate matches exactly the lower bound on the performance of any black-box method on the considered problem class. Additionally, we propose an adaptive Polyak method with a momentum term, where the step sizes are independent of the number of iterates. We establish that the algorithm also attains the optimal convergence rate. We investigate the alternating projection method. We derive a convergence rate $\left( \frac{2N }{ 2N+1 } \right)^N\tfrac{R}{\sqrt{2N+1}}$ for the last iterate, where $R$ is a bound on the distance between the initial iterate and a solution. An example is also provided to illustrate the exactness of the rate.