Optimal inexactness schedules for Tunable Oracle based Methods

TL;DR

This paper develops optimal schedules for controlling the inexactness of oracles in first-order optimization methods, balancing convergence speed and computational cost, with theoretical guarantees and practical experiments.

Contribution

It introduces a framework for choosing inexactness levels dynamically to optimize convergence and computational efficiency in various optimization algorithms.

Findings

Optimal inexactness schedules improve convergence efficiency.

Tuning oracle precision yields better trade-offs than constant precision.

Numerical experiments validate the practical benefits of the proposed approach.

Abstract

Several recent works address the impact of inexact oracles in the convergence analysis of modern first-order optimization techniques, e.g. Bregman Proximal Gradient and Prox-Linear methods as well as their accelerated variants, extending their field of applicability. In this paper, we consider situations where the oracle's inexactness can be chosen upon demand, more precision coming at a computational price counterpart. Our main motivations arise from oracles requiring the solving of auxiliary subproblems or the inexact computation of involved quantities, e.g. a mini-batch stochastic gradient as a full-gradient estimate. We propose optimal inexactness schedules according to presumed oracle cost models and patterns of worst-case guarantees, covering among others convergence results of the aforementioned methods under the presence of inexactness. Specifically, we detail how to choose the level of inexactness at each iteration to obtain the best trade-off between convergence and computational investments. Furthermore, we highlight the benefits one can expect by tuning those oracles' quality instead of keeping it constant throughout. Finally, we provide extensive numerical experiments that support the practical interest of our approach, both in offline and online settings, applied to the Fast Gradient algorithm.