Hierarchical reinforcement learning for in-hand robotic manipulation using Davenport chained rotations

TL;DR

This paper introduces a hierarchical reinforcement learning approach using Davenport chained rotations to decompose complex 3D rotation tasks into simpler skills, reducing training time and improving success rates in robotic manipulation.

Contribution

It proposes a novel decomposition method for 3D rotations in reinforcement learning, enabling more efficient training of robotic manipulation policies.

Findings

Decomposition improves success rates by ~10% on complex rotations.

Hierarchical approach reduces training data requirements.

Performance is comparable or better than end-to-end methods under resource constraints.

Abstract

End-to-end reinforcement learning techniques are among the most successful methods for robotic manipulation tasks. However, the training time required to find a good policy capable of solving complex tasks is prohibitively large. Therefore, depending on the computing resources available, it might not be feasible to use such techniques. The use of domain knowledge to decompose manipulation tasks into primitive skills, to be performed in sequence, could reduce the overall complexity of the learning problem, and hence reduce the amount of training required to achieve dexterity. In this paper, we propose the use of Davenport chained rotations to decompose complex 3D rotation goals into a concatenation of a smaller set of more simple rotation skills. State-of-the-art reinforcement-learning-based methods can then be trained using less overall simulated experience. We compare its performance with the popular Hindsight Experience Replay method, trained in an end-to-end fashion using the same amount of experience in a simulated robotic hand environment. Despite a general decrease in performance of the primitive skills when being sequentially executed, we find that decomposing arbitrary 3D rotations into elementary rotations is beneficial when computing resources are limited, obtaining increases of success rates of approximately 10% on the most complex 3D rotations with respect to the success rates obtained by HER trained in an end-to-end fashion, and increases of success rates between 20% and 40% on the most simple rotations.