Robotic Lever Manipulation using Hindsight Experience Replay and Shapley Additive Explanations

TL;DR

This paper presents a method for robotic lever manipulation using deep reinforcement learning with sparse rewards, transferring policies from simulation to real-world, and explaining decisions with SHAP to enhance interpretability.

Contribution

It introduces a combined approach of DRL with Hindsight Experience Replay and SHAP explanations for robotic control, addressing simulation-to-real transfer and interpretability challenges.

Findings

Successful transfer of policies from PyBullet and Gazebo simulators to real robot

SHAP explanations reveal intuitive and non-intuitive decision factors

Demonstrates the feasibility of sparse reward learning in complex robotic tasks

Abstract

This paper deals with robotic lever control using Explainable Deep Reinforcement Learning. First, we train a policy by using the Deep Deterministic Policy Gradient algorithm and the Hindsight Experience Replay technique, where the goal is to control a robotic manipulator to manipulate a lever. This enables us both to use continuous states and actions and to learn with sparse rewards. Being able to learn from sparse rewards is especially desirable for Deep Reinforcement Learning because designing a reward function for complex tasks such as this is challenging. We first train in the PyBullet simulator, which accelerates the training procedure, but is not accurate on this task compared to the real-world environment. After completing the training in PyBullet, we further train in the Gazebo simulator, which runs more slowly than PyBullet, but is more accurate on this task. We then transfer the policy to the real-world environment, where it achieves comparable performance to the simulated environments for most episodes. To explain the decisions of the policy we use the SHAP method to create an explanation model based on the episodes done in the real-world environment. This gives us some results that agree with intuition, and some that do not. We also question whether the independence assumption made when approximating the SHAP values influences the accuracy of these values for a system such as this, where there are some correlations between the states.