Multitask Adaptation by Retrospective Exploration with Learned World Models

TL;DR

This paper introduces RAMa, a meta-learned addressing model that enhances model-based reinforcement learning by reusing past task data for retrospective exploration, leading to faster learning across diverse environments.

Contribution

The paper presents RAMa, a novel meta-learned addressing model that improves sample efficiency in MBRL by leveraging a continuously growing task-agnostic storage for retrospective exploration.

Findings

RAMa accelerates learning in multiple domains.

Retrospective exploration improves dynamics modeling.

Effective in photorealistic environments.

Abstract

Model-based reinforcement learning (MBRL) allows solving complex tasks in a sample-efficient manner. However, no information is reused between the tasks. In this work, we propose a meta-learned addressing model called RAMa that provides training samples for the MBRL agent taken from continuously growing task-agnostic storage. The model is trained to maximize the expected agent's performance by selecting promising trajectories solving prior tasks from the storage. We show that such retrospective exploration can accelerate the learning process of the MBRL agent by better informing learned dynamics and prompting agent with exploratory trajectories. We test the performance of our approach on several domains from the DeepMind control suite, from Metaworld multitask benchmark, and from our bespoke environment implemented with a robotic NVIDIA Isaac simulator to test the ability of the model to act in a photorealistic, ray-traced environment.