Learning in Deep Neural Networks Using a Biologically Inspired Optimizer

TL;DR

This paper introduces GRAPES, a biologically inspired optimizer for neural networks that improves convergence, accuracy, scalability, and mitigates catastrophic forgetting by incorporating synaptic integration principles observed in the brain.

Contribution

The paper proposes GRAPES, a novel optimizer inspired by neurophysiology, which enhances neural network training by modulating error signals based on weight distribution.

Findings

Improves convergence rate of neural networks.

Enhances classification accuracy for ANNs and SNNs.

Supports scalability and reduces catastrophic forgetting.

Abstract

Plasticity circuits in the brain are known to be influenced by the distribution of the synaptic weights through the mechanisms of synaptic integration and local regulation of synaptic strength. However, the complex interplay of stimulation-dependent plasticity with local learning signals is disregarded by most of the artificial neural network training algorithms devised so far. Here, we propose a novel biologically inspired optimizer for artificial (ANNs) and spiking neural networks (SNNs) that incorporates key principles of synaptic integration observed in dendrites of cortical neurons: GRAPES (Group Responsibility for Adjusting the Propagation of Error Signals). GRAPES implements a weight-distribution dependent modulation of the error signal at each node of the neural network. We show that this biologically inspired mechanism leads to a systematic improvement of the convergence rate of the network, and substantially improves classification accuracy of ANNs and SNNs with both feedforward and recurrent architectures. Furthermore, we demonstrate that GRAPES supports performance scalability for models of increasing complexity and mitigates catastrophic forgetting by enabling networks to generalize to unseen tasks based on previously acquired knowledge. The local characteristics of GRAPES minimize the required memory resources, making it optimally suited for dedicated hardware implementations. Overall, our work indicates that reconciling neurophysiology insights with machine intelligence is key to boosting the performance of neural networks.