Affine-invariant contracting-point methods for Convex Optimization

TL;DR

This paper introduces affine-invariant Contracting-Point methods for convex optimization, unifying and extending existing algorithms with new complexity bounds and practical implementations, including a second-order scheme with promising numerical results.

Contribution

The paper develops a general affine-invariant framework for Contracting-Point methods, connecting tensor methods with classical algorithms and providing new complexity bounds and practical schemes.

Findings

Global convergence rate of ${ m O}(1 / k^p)$ for tensor degree p

Recovery of Frank-Wolfe algorithm as a special case

Efficient inexact second-order Contracting Newton method with good practical performance

Abstract

In this paper, we develop new affine-invariant algorithms for solving composite convex minimization problems with bounded domain. We present a general framework of Contracting-Point methods, which solve at each iteration an auxiliary subproblem restricting the smooth part of the objective function onto contraction of the initial domain. This framework provides us with a systematic way for developing optimization methods of different order, endowed with the global complexity bounds. We show that using an appropriate affine-invariant smoothness condition, it is possible to implement one iteration of the Contracting-Point method by one step of the pure tensor method of degree $p \geq 1$. The resulting global rate of convergence in functional residual is then ${\cal O}(1 / k^p)$, where $k$ is the iteration counter. It is important that all constants in our bounds are affine-invariant. For $p = 1$, our scheme recovers well-known Frank-Wolfe algorithm, providing it with a new interpretation by a general perspective of tensor methods. Finally, within our framework, we present efficient implementation and total complexity analysis of the inexact second-order scheme $(p = 2)$, called Contracting Newton method. It can be seen as a proper implementation of the trust-region idea. Preliminary numerical results confirm its good practical performance both in the number of iterations, and in computational time.