General policy mapping: online continual reinforcement learning inspired on the insect brain

TL;DR

This paper introduces a biologically inspired model for online continual reinforcement learning that leverages shared policy layers and offline feature training, enabling positive backward transfer and efficient learning in resource-limited settings.

Contribution

It proposes a novel RL model inspired by insect brains that combines offline feature extraction with shared policy layers for improved online learning.

Findings

Positive backward transfer observed across tasks

Biologically inspired network restrictions are crucial for convergence

Model enables efficient online RL in resource-constrained environments

Abstract

We have developed a model for online continual or lifelong reinforcement learning (RL) inspired on the insect brain. Our model leverages the offline training of a feature extraction and a common general policy layer to enable the convergence of RL algorithms in online settings. Sharing a common policy layer across tasks leads to positive backward transfer, where the agent continuously improved in older tasks sharing the same underlying general policy. Biologically inspired restrictions to the agent's network are key for the convergence of RL algorithms. This provides a pathway towards efficient online RL in resource-constrained scenarios.