Feedback Control for Online Training of Neural Networks

TL;DR

This paper introduces a novel feedback control strategy for online training of CNNs that adaptively adjusts the learning rate based on network performance, leading to faster and more robust training.

Contribution

It proposes E/PD-Control, a performance-based learning rate adaptation method combining PD control with exponential boosting, with proven stability and improved training results.

Findings

Faster accuracy growth on CIFAR-10 and Fashion-MNIST datasets

Higher final network accuracy levels achieved

Robustness to parameter variations

Abstract

Convolutional neural networks (CNNs) are commonly used for image classification tasks, raising the challenge of their application on data flows. During their training, adaptation is often performed by tuning the learning rate. Usual learning rate strategies are time-based i.e. monotonously decreasing. In this paper, we advocate switching to a performance-based adaptation, in order to improve the learning efficiency. We present E (Exponential)/PD (Proportional Derivative)-Control, a conditional learning rate strategy that combines a feedback PD controller based on the CNN loss function, with an exponential control signal to smartly boost the learning and adapt the PD parameters. Stability proof is provided as well as an experimental evaluation using two state of the art image datasets (CIFAR-10 and Fashion-MNIST). Results show better performances than the related works (faster network accuracy growth reaching higher levels) and robustness of the E/PD-Control regarding its parametrization.