Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations

TL;DR

This paper introduces POSG, a novel reinforcement learning algorithm that uses state-only demonstrations to improve learning efficiency and control performance in sparse-reward environments, reducing reliance on high-quality action data.

Contribution

The paper proposes POSG, an efficient method leveraging state-only demonstrations with a trajectory importance mechanism to guide policy optimization in sparse-reward settings.

Findings

POSG outperforms baselines in control performance.

Faster convergence in four benchmark environments.

Effective use of state-only demonstrations for guidance.

Abstract

The sparsity of reward feedback remains a challenging problem in online deep reinforcement learning (DRL). Previous approaches have utilized offline demonstrations to achieve impressive results in multiple hard tasks. However, these approaches place high demands on demonstration quality, and obtaining expert-like actions is often costly and unrealistic. To tackle these problems, we propose a simple and efficient algorithm called Policy Optimization with Smooth Guidance (POSG), which leverages a small set of state-only demonstrations (where expert action information is not included in demonstrations) to indirectly make approximate and feasible long-term credit assignments and facilitate exploration. Specifically, we first design a trajectory-importance evaluation mechanism to determine the quality of the current trajectory against demonstrations. Then, we introduce a guidance reward computation technology based on trajectory importance to measure the impact of each state-action pair, fusing the demonstrator's state distribution with reward information into the guidance reward. We theoretically analyze the performance improvement caused by smooth guidance rewards and derive a new worst-case lower bound on the performance improvement. Extensive results demonstrate POSG's significant advantages in control performance and convergence speed in four sparse-reward environments, including the grid-world maze, Hopper-v4, HalfCheetah-v4, and Ant maze. Notably, the specific metrics and quantifiable results are investigated to demonstrate the superiority of POSG.