Iterative Model-Based Reinforcement Learning Using Simulations in the Differentiable Neural Computer

TL;DR

This paper introduces a lifelong learning framework where a reinforcement learning agent improves its policy through iterative training in simulated environments generated by a Differentiable Neural Computer, enabling continual learning from pixels.

Contribution

It presents the Neural Computer Agent (NCA), a novel architecture combining RL with a DNC-based environment model for iterative policy improvement and continual learning.

Findings

DNC models can learn from pixels to simulate new tasks.

Agents trained entirely in simulation using PPO perform well on real tasks.

The approach enables continual learning across multiple tasks.

Abstract

We propose a lifelong learning architecture, the Neural Computer Agent (NCA), where a Reinforcement Learning agent is paired with a predictive model of the environment learned by a Differentiable Neural Computer (DNC). The agent and DNC model are trained in conjunction iteratively. The agent improves its policy in simulations generated by the DNC model and rolls out the policy to the live environment, collecting experiences in new portions or tasks of the environment for further learning. Experiments in two synthetic environments show that DNC models can continually learn from pixels alone to simulate new tasks as they are encountered by the agent, while the agents can be successfully trained to solve the tasks using Proximal Policy Optimization entirely in simulations.