Derivation of Coordinate Descent Algorithms from Optimal Control Theory

TL;DR

This paper demonstrates how coordinate descent algorithms can be systematically derived from optimal control theory, linking their convergence to Lyapunov functions and the Hessian of the objective.

Contribution

It introduces a novel derivation of coordinate descent algorithms from optimal control principles using Lyapunov functions and maximum principles.

Findings

Coordinate descent algorithms can be derived from optimal control theory.

Convergence is linked to Lyapunov function dissipation.

Hessian of the objective guides the search metric.

Abstract

Recently, it was posited that disparate optimization algorithms may be coalesced in terms of a central source emanating from optimal control theory. Here we further this proposition by showing how coordinate descent algorithms may be derived from this emerging new principle. In particular, we show that basic coordinate descent algorithms can be derived using a maximum principle and a collection of max functions as "control" Lyapunov functions. The convergence of the resulting coordinate descent algorithms is thus connected to the controlled dissipation of their corresponding Lyapunov functions. The operational metric for the search vector in all cases is given by the Hessian of the convex objective function.