Reinforcement Learning in a Neurally Controlled Robot Using Dopamine Modulated STDP

TL;DR

This paper demonstrates how dopamine-modulated STDP in spiking neural networks enables a robot to learn, unlearn, and adapt to changing reinforcement tasks in complex environments, highlighting biological plausibility and practical capabilities.

Contribution

It introduces a novel implementation of dopamine-modulated STDP in an embodied robot, showing effective learning and adaptation in dynamic reinforcement learning scenarios.

Findings

Robot learned food-attraction in all trials

Successfully unlearned behaviors when environment changed

Achieved 95% success in complex food-container task

Abstract

Recent work has shown that dopamine-modulated STDP can solve many of the issues associated with reinforcement learning, such as the distal reward problem. Spiking neural networks provide a useful technique in implementing reinforcement learning in an embodied context as they can deal with continuous parameter spaces and as such are better at generalizing the correct behaviour to perform in a given context. In this project we implement a version of DA-modulated STDP in an embodied robot on a food foraging task. Through simulated dopaminergic neurons we show how the robot is able to learn a sequence of behaviours in order to achieve a food reward. In tests the robot was able to learn food-attraction behaviour, and subsequently unlearn this behaviour when the environment changed, in all 50 trials. Moreover we show that the robot is able to operate in an environment whereby the optimal behaviour changes rapidly and so the agent must constantly relearn. In a more complex environment, consisting of food-containers, the robot was able to learn food-container attraction in 95% of trials, despite the large temporal distance between the correct behaviour and the reward. This is achieved by shifting the dopamine response from the primary stimulus (food) to the secondary stimulus (food-container). Our work provides insights into the reasons behind some observed biological phenomena, such as the bursting behaviour observed in dopaminergic neurons. As well as demonstrating how spiking neural network controlled robots are able to solve a range of reinforcement learning tasks.