Improving Energy Conserving Descent for Machine Learning: Theory and Practice

TL;DR

This paper introduces ECDSep, a novel energy-conserving optimization algorithm inspired by physical dynamical systems, demonstrating competitive performance in machine learning tasks and offering theoretical insights into its dynamics.

Contribution

The paper develops the theory of Energy Conserving Descent and introduces ECDSep, a new gradient-based optimizer that improves performance and simplifies hyper-parameter tuning for diverse problems.

Findings

ECDSep achieves competitive or superior results compared to SGD, Adam, and AdamW.

The method offers better control over the optimization dynamics.

Limitations suggest avenues for further enhancement.

Abstract

We develop the theory of Energy Conserving Descent (ECD) and introduce ECDSep, a gradient-based optimization algorithm able to tackle convex and non-convex optimization problems. The method is based on the novel ECD framework of optimization as physical evolution of a suitable chaotic energy-conserving dynamical system, enabling analytic control of the distribution of results - dominated at low loss - even for generic high-dimensional problems with no symmetries. Compared to previous realizations of this idea, we exploit the theoretical control to improve both the dynamics and chaos-inducing elements, enhancing performance while simplifying the hyper-parameter tuning of the optimization algorithm targeted to different classes of problems. We empirically compare with popular optimization methods such as SGD, Adam and AdamW on a wide range of machine learning problems, finding competitive or improved performance compared to the best among them on each task. We identify limitations in our analysis pointing to possibilities for additional improvements.