Automating Reinforcement Learning with Example-based Resets

TL;DR

This paper introduces an autonomous reset mechanism in reinforcement learning using an additional agent that learns to reset, reducing manual interventions in real-world robotic tasks and enabling more autonomous learning.

Contribution

It presents a novel self-supervised reset agent that learns to trigger resets, facilitating autonomous reinforcement learning in robotics without manual resets.

Findings

Reset agent reduces manual resets in tasks

Forward policy improves over time with autonomous resets

Method works on both simulated and real-world tasks

Abstract

Deep reinforcement learning has enabled robots to learn motor skills from environmental interactions with minimal to no prior knowledge. However, existing reinforcement learning algorithms assume an episodic setting, in which the agent resets to a fixed initial state distribution at the end of each episode, to successfully train the agents from repeated trials. Such reset mechanism, while trivial for simulated tasks, can be challenging to provide for real-world robotics tasks. Resets in robotic systems often require extensive human supervision and task-specific workarounds, which contradicts the goal of autonomous robot learning. In this paper, we propose an extension to conventional reinforcement learning towards greater autonomy by introducing an additional agent that learns to reset in a self-supervised manner. The reset agent preemptively triggers a reset to prevent manual resets and implicitly imposes a curriculum for the forward agent. We apply our method to learn from scratch on a suite of simulated and real-world continuous control tasks and demonstrate that the reset agent successfully learns to reduce manual resets whilst also allowing the forward policy to improve gradually over time.