Meta-Reinforcement Learning with Self-Modifying Networks

TL;DR

This paper introduces MetODS, a meta-reinforcement learning system with self-modifying networks that adapt dynamically, enabling rapid learning, generalization, and flexible control in complex environments.

Contribution

It presents a novel meta-reinforcement learning model with self-modifying, dynamic synapses inspired by biological plasticity, allowing continual learning and adaptation beyond traditional fixed networks.

Findings

Enables one-shot learning with a single-layer dynamic synapse network.

Generalizes navigation principles to unseen environments.

Demonstrates strong adaptive motor policy learning.

Abstract

Deep Reinforcement Learning has demonstrated the potential of neural networks tuned with gradient descent for solving complex tasks in well-delimited environments. However, these neural systems are slow learners producing specialized agents with no mechanism to continue learning beyond their training curriculum. On the contrary, biological synaptic plasticity is persistent and manifold, and has been hypothesized to play a key role in executive functions such as working memory and cognitive flexibility, potentially supporting more efficient and generic learning abilities. Inspired by this, we propose to build networks with dynamic weights, able to continually perform self-reflexive modification as a function of their current synaptic state and action-reward feedback, rather than a fixed network configuration. The resulting model, MetODS (for Meta-Optimized Dynamical Synapses) is a broadly applicable meta-reinforcement learning system able to learn efficient and powerful control rules in the agent policy space. A single layer with dynamic synapses can perform one-shot learning, generalizes navigation principles to unseen environments and manifests a strong ability to learn adaptive motor policies.