Offline Robot Reinforcement Learning with Uncertainty-Guided Human Expert Sampling

TL;DR

This paper introduces an uncertainty-guided method for incorporating human demonstrations into offline reinforcement learning, significantly improving sample efficiency and policy performance in robotics tasks.

Contribution

It proposes a novel uncertainty-based approach to selectively inject human demonstration data into offline RL training, enhancing sample efficiency and policy quality.

Findings

Outperforms naive expert data integration methods

Reduces sample complexity in offline RL

Effective in MuJoCo and OffWorld Gym environments

Abstract

Recent advances in batch (offline) reinforcement learning have shown promising results in learning from available offline data and proved offline reinforcement learning to be an essential toolkit in learning control policies in a model-free setting. An offline reinforcement learning algorithm applied to a dataset collected by a suboptimal non-learning-based algorithm can result in a policy that outperforms the behavior agent used to collect the data. Such a scenario is frequent in robotics, where existing automation is collecting operational data. Although offline learning techniques can learn from data generated by a sub-optimal behavior agent, there is still an opportunity to improve the sample complexity of existing offline reinforcement learning algorithms by strategically introducing human demonstration data into the training process. To this end, we propose a novel approach that uses uncertainty estimation to trigger the injection of human demonstration data and guide policy training towards optimal behavior while reducing overall sample complexity. Our experiments show that this approach is more sample efficient when compared to a naive way of combining expert data with data collected from a sub-optimal agent. We augmented an existing offline reinforcement learning algorithm Conservative Q-Learning with our approach and performed experiments on data collected from MuJoCo and OffWorld Gym learning environments.