Deep Learning via Neural Energy Descent

TL;DR

This paper introduces Neural Energy Descent (NED), a PDE-based framework that reformulates deep learning as an energy minimization problem, offering a novel interpretation and computational advantages over traditional methods like SGD.

Contribution

The paper presents a new PDE-based approach to deep learning optimization, providing a global energy minimization perspective and demonstrating computational efficiency.

Findings

NED can efficiently solve deep learning problems via PDE evolution.

Numerical results show NED outperforms stochastic gradient descent in certain scenarios.

NED offers a novel theoretical interpretation of deep learning as energy minimization.

Abstract

This paper proposes the Nerual Energy Descent (NED) via neural network evolution equations for a wide class of deep learning problems. We show that deep learning can be reformulated as the evolution of network parameters in an evolution equation and the steady state solution of the partial differential equation (PDE) provides a solution to deep learning. This equation corresponds to a gradient descent flow of a variational problem and hence the proposed time-dependent PDE solves an energy minimization problem to obtain a global minimizer of deep learning. This gives a novel interpretation and solution to deep learning optimization. The computational complexity of the proposed energy descent method can be enhanced by randomly sampling the spatial domain of the PDE leading to an efficient NED. Numerical examples are provided to demonstrate the numerical advantage of NED over stochastic gradient descent (SGD).