Composing Option Sequences by Adaptation: Initial Results

TL;DR

This paper introduces a framework and methods for adapting deep reinforcement learning options to successfully compose novel task sequences in robot manipulation, addressing the challenge of sequence failure in real-world settings.

Contribution

The paper presents a new framework and three adaptation approaches to improve the success rate of composing deep RL options in novel sequences.

Findings

The framework can predict sequence success before execution.

Adaptation methods improve sequence success in robot pick-and-place tasks.

Training options to reach specific points enhances sequence compatibility.

Abstract

Robot manipulation in real-world settings often requires adapting the robot's behavior to the current situation, such as by changing the sequences in which policies execute to achieve the desired task. Problematically, however, we show that composing a novel sequence of five deep RL options to perform a pick-and-place task is unlikely to successfully complete, even if their initiation and termination conditions align. We propose a framework to determine whether sequences will succeed a priori, and examine three approaches that adapt options to sequence successfully if they will not. Crucially, our adaptation methods consider the actual subset of points that the option is trained from or where it ends: (1) trains the second option to start where the first ends; (2) trains the first option to reach the centroid of where the second starts; and (3) trains the first option to reach the median of where the second starts. Our results show that our framework and adaptation methods have promise in adapting options to work in novel sequences.