Efficiently Training On-Policy Actor-Critic Networks in Robotic Deep Reinforcement Learning with Demonstration-like Sampled Exploration

TL;DR

This paper introduces a framework that enhances on-policy actor-critic reinforcement learning in robotics by integrating demonstration data through a clustering-based approach, significantly improving sample efficiency and convergence speed.

Contribution

It presents a novel method combining demonstration-guided exploration with on-policy RL using clustering, which is more effective in high-dimensional robotic environments.

Findings

Sample efficiency improved by 20-40%.

Accelerated convergence in complex robotic tasks.

Achieved higher final rewards with demonstration integration.

Abstract

In complex environments with high dimension, training a reinforcement learning (RL) model from scratch often suffers from lengthy and tedious collection of agent-environment interactions. Instead, leveraging expert demonstration to guide RL agent can boost sample efficiency and improve final convergence. In order to better integrate expert prior with on-policy RL models, we propose a generic framework for Learning from Demonstration (LfD) based on actor-critic algorithms. Technically, we first employ K-Means clustering to evaluate the similarity of sampled exploration with demonstration data. Then we increase the likelihood of actions in similar frames by modifying the gradient update strategy to leverage demonstration. We conduct experiments on 4 standard benchmark environments in Mujoco and 2 self-designed robotic environments. Results show that, under certain condition, our algorithm can improve sample efficiency by 20% ~ 40%. By combining our framework with on-policy algorithms, RL models can accelerate convergence and obtain better final mean episode rewards especially in complex robotic context where interactions are expensive.