Dynamical loss functions shape landscape topography and improve learning in artificial neural networks

TL;DR

This paper introduces dynamical loss functions that oscillate during training, reshaping the loss landscape to enhance neural network learning and validation accuracy without changing the global minima.

Contribution

It presents a novel class of dynamical loss functions derived from standard ones, demonstrating their ability to improve training outcomes and landscape exploration.

Findings

Dynamical loss functions improve validation accuracy.

Oscillations reshape the loss landscape beneficially.

Landscape evolution shows edge-of-instability phenomena.

Abstract

Dynamical loss functions are derived from standard loss functions used in supervised classification tasks, but are modified so that the contribution from each class periodically increases and decreases. These oscillations globally alter the loss landscape without affecting the global minima. In this paper, we demonstrate how to transform cross-entropy and mean squared error into dynamical loss functions. We begin by discussing the impact of increasing the size of the neural network or the learning rate on the depth and sharpness of the minima that the system explores. Building on this intuition, we propose several versions of dynamical loss functions and use a simple classification problem where we can show how they significantly improve validation accuracy for networks of varying sizes. Finally, we explore how the landscape of these dynamical loss functions evolves during training, highlighting the emergence of instabilities that may be linked to edge-of-instability minimization.