Lifelong Reinforcement Learning via Neuromodulation

TL;DR

This paper proposes a new framework inspired by neuroscience to develop adaptive reinforcement learning algorithms capable of lifelong learning across multiple tasks and changing environments.

Contribution

It introduces an abstract framework integrating neuroscience theories into reinforcement learning, exemplified by a neuromodulation-based algorithm tested on non-stationary bandit problems.

Findings

The neuromodulation-inspired algorithm adapts effectively to non-stationary environments.

Empirical validation shows improved performance over traditional methods.

Framework links reinforcement learning with neuroscience insights.

Abstract

Navigating multiple tasks$\unicode{x2014}$for instance in succession as in continual or lifelong learning, or in distributions as in meta or multi-task learning$\unicode{x2014}$requires some notion of adaptation. Evolution over timescales of millennia has imbued humans and other animals with highly effective adaptive learning and decision-making strategies. Central to these functions are so-called neuromodulatory systems. In this work we introduce an abstract framework for integrating theories and evidence from neuroscience and the cognitive sciences into the design of adaptive artificial reinforcement learning algorithms. We give a concrete instance of this framework built on literature surrounding the neuromodulators Acetylcholine (ACh) and Noradrenaline (NA), and empirically validate the effectiveness of the resulting adaptive algorithm in a non-stationary multi-armed bandit problem. We conclude with a theory-based experiment proposal providing an avenue to link our framework back to efforts in experimental neuroscience.