Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity

TL;DR

This paper introduces a novel method for training neural networks with differentiable neuromodulated plasticity, inspired by biological brain mechanisms, leading to improved performance in reinforcement and supervised learning tasks.

Contribution

It presents the first gradient descent training method for neuromodulated plasticity in neural networks, extending differentiable Hebbian plasticity with a new neuromodulation formulation.

Findings

Neuromodulated plastic LSTMs outperform standard LSTMs on language modeling.

The approach enhances lifelong learning capabilities in neural networks.

Differentiable neuromodulation improves performance across tasks.

Abstract

The impressive lifelong learning in animal brains is primarily enabled by plastic changes in synaptic connectivity. Importantly, these changes are not passive, but are actively controlled by neuromodulation, which is itself under the control of the brain. The resulting self-modifying abilities of the brain play an important role in learning and adaptation, and are a major basis for biological reinforcement learning. Here we show for the first time that artificial neural networks with such neuromodulated plasticity can be trained with gradient descent. Extending previous work on differentiable Hebbian plasticity, we propose a differentiable formulation for the neuromodulation of plasticity. We show that neuromodulated plasticity improves the performance of neural networks on both reinforcement learning and supervised learning tasks. In one task, neuromodulated plastic LSTMs with millions of parameters outperform standard LSTMs on a benchmark language modeling task (controlling for the number of parameters). We conclude that differentiable neuromodulation of plasticity offers a powerful new framework for training neural networks.