On the Effectiveness of the z-Transform Method in Quadratic Optimization

TL;DR

This paper explores the use of the z-transform method to analyze the asymptotic behavior of various optimization algorithms, demonstrating its effectiveness in deriving new theoretical insights.

Contribution

It introduces the application of the z-transform to quadratic optimization, extending classical signal processing tools to analyze convergence of optimization algorithms.

Findings

Z-transform characterizes convergence in gradient descent.

Extension of analysis to Nesterov acceleration and stochastic methods.

Demonstrates versatility of z-transform in optimization theory.

Abstract

The z-transform of a sequence is a classical tool used within signal processing, control theory, computer science, and electrical engineering. It allows for studying sequences from their generating functions, with many operations that can be equivalently defined on the original sequence and its $z$-transform. In particular, the z-transform method focuses on asymptotic behaviors and allows the use of Taylor expansions. We present a sequence of results of increasing significance and difficulty for linear models and optimization algorithms, demonstrating the effectiveness and versatility of the z-transform method in deriving new asymptotic results. Starting from the simplest gradient descent iterations in an infinite-dimensional Hilbert space, we show how the spectral dimension characterizes the convergence behavior. We then extend the analysis to Nesterov acceleration, averaging techniques, and stochastic gradient descent.