Tannenbaum's gain-margin optimization meets Polyak's heavy-ball algorithm

TL;DR

This paper connects optimization algorithms with robust control theory, revealing new insights into their performance limits and proposing methods to achieve faster convergence through feedback regulation techniques.

Contribution

It establishes a novel link between Polyak's heavy-ball algorithm and gain margin optimization in control theory, using Nevanlinna--Pick interpolation.

Findings

Polyak's heavy-ball optimality is linked to gain margin optimization.

Periodic algorithms cannot surpass certain convergence rates due to transmission zeros.

Implicit algorithms with feedback regulation can achieve better convergence rates.

Abstract

This paper highlights an apparent, yet relatively unknown link between algorithm design in optimization theory and controller synthesis in robust control. Specifically, quadratic optimization can be recast as a regulation problem within the framework of $\mathcal{H}_\infty$ control. From this vantage point, the optimality of Polyak's fastest heavy-ball algorithm can be ascertained as a solution to a gain margin optimization problem. The approach is independent of Polyak's original and brilliant argument, and relies on foundational work by Tannenbaum, who introduced and solved gain margin optimization via Nevanlinna--Pick interpolation theory. The link between first-order optimization methods and robust control sheds new light on the limits of algorithmic performance of such methods, and suggests a framework where similar computational tasks can be systematically studied and algorithms optimized. In particular, it raises the question as to whether periodically scheduled algorithms can achieve faster rates for quadratic optimization, in a manner analogous to periodic control that extends the gain margin beyond that of time-invariant control. This turns out not to be the case, due to the analytic obstruction of a transmission zero that is inherent in causal schemes. Interestingly, this obstruction can be removed with implicit algorithms, cast as feedback regulation problems with causal, but not strictly causal dynamics, thereby devoid of the transmission zero at infinity and able to achieve superior convergence rates.