On The Mathematics of the Natural Physics of Optimization

TL;DR

This paper develops a new mathematical physics framework for understanding and deriving optimization algorithms based on non-Newtonian dynamics and optimal control theory.

Contribution

It introduces a novel theory linking optimization algorithms to universal physical laws and control principles, explaining existing algorithms and generating new ones.

Findings

Many algorithms can be derived from the proposed physics-based framework.

The theory explains the optimality conditions through a natural vector field.

Applications demonstrate the generation and explanation of various algorithms.

Abstract

A number of optimization algorithms have been inspired by the physics of Newtonian motion. Here, we ask the question: do algorithms themselves obey some ``natural laws of motion,'' and can they be derived by an application of these laws? We explore this question by positing the theory that optimization algorithms may be considered as some manifestation of hidden algorithm primitives that obey certain universal non-Newtonian dynamics. This natural physics of optimization is developed by equating the terminal transversality conditions of an optimal control problem to the generalized Karush/John-Kuhn-Tucker conditions of an optimization problem. Through this equivalence formulation, the data functions of a given constrained optimization problem generate a natural vector field that permeates an entire hidden space with information on the optimality conditions. An ``action-at-a-distance'' operation via a Pontryagin-type minimum principle produces a local action to deliver a globalized result by way of a Hamilton-Jacobi inequality. An inverse-optimal algorithm is generated by performing control jumps that dissipate quantized ``energy'' defined by a search Lyapunov function. Illustrative applications of the proposed theory show that a large number of algorithms can be generated and explained in terms of the new mathematical physics of optimization.