# Prefrontal meta-control incorporating mental simulation enhances the adaptivity of reinforcement learning agents in dynamic environments

## TL;DR

This paper introduces Meta-Dyna, a brain-inspired reinforcement learning system that adapts quickly to changing environments by mimicking prefrontal and hippocampal brain functions.

## Contribution

The novel Meta-Dyna architecture integrates prefrontal meta-control and hippocampal replay to enhance adaptivity in reinforcement learning.

## Key findings

- Meta-Dyna outperformed baseline algorithms in average reward and choice optimality.
- The system achieved success with fewer trials in dynamic and uncertain environments.
- Performance was validated across three distinct reinforcement learning paradigms.

## Abstract

Recent advances in computational neuroscience highlight the significance of prefrontal cortical meta-control mechanisms in facilitating flexible and adaptive human behavior. In addition, hippocampal function, particularly mental simulation capacity, proves essential in this adaptive process. Rooted from these neuroscientific insights, we present Meta-Dyna, a novel neuroscience-inspired reinforcement learning architecture that demonstrates rapid adaptation to environmental dynamics whilst managing variable goal states and state-transition uncertainties.

This architectural framework implements prefrontal meta-control mechanisms integrated with hippocampal replay function, which in turn optimized task performance with limited experiences. We evaluated this approach through comprehensive experimental simulations across three distinct paradigms: the two-stage Markov decision task, which frequently serves in human learning and decision-making research; stochastic GridWorldLoCA, an established benchmark suite for model-based reinforcement learning; and a stochastic Atari Pong variant incorporating multiple goals under uncertainty.

Experimental results demonstrate Meta-Dyna's superior performance compared with baseline reinforcement learning algorithms across multiple metrics: average reward, choice optimality, and a number of trials for success.

These findings advance our understanding of computational reinforcement learning whilst contributing to the development of brain-inspired learning agents capable of flexible, goal-directed behavior within dynamic environments.