Evaluating task-agnostic exploration for fixed-batch learning of arbitrary future tasks

TL;DR

This paper evaluates task-agnostic exploration methods for offline learning of diverse robotic tasks using fixed datasets, demonstrating their effectiveness in simulation and real-world robot control without further interaction.

Contribution

It provides a comprehensive evaluation of exploration strategies for offline reinforcement learning in robotics, highlighting their potential for learning multiple tasks from fixed datasets.

Findings

Exploration methods enable effective offline learning in robotics.

Success demonstrated on simulation and real robot arm tasks.

Provides open-source code for reproducibility.

Abstract

Deep reinforcement learning has been shown to solve challenging tasks where large amounts of training experience is available, usually obtained online while learning the task. Robotics is a significant potential application domain for many of these algorithms, but generating robot experience in the real world is expensive, especially when each task requires a lengthy online training procedure. Off-policy algorithms can in principle learn arbitrary tasks from a diverse enough fixed dataset. In this work, we evaluate popular exploration methods by generating robotics datasets for the purpose of learning to solve tasks completely offline without any further interaction in the real world. We present results on three popular continuous control tasks in simulation, as well as continuous control of a high-dimensional real robot arm. Code documenting all algorithms, experiments, and hyper-parameters is available at https://github.com/qutrobotlearning/batchlearning.