Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Environments with Delayed Rewards

TL;DR

This paper enhances Proximal Policy Optimization for reinforcement learning with delayed rewards by integrating offline expert data and temporal logic-based reward shaping, leading to faster and more effective learning.

Contribution

It introduces a hybrid PPO architecture with reward shaping via TWTL, providing theoretical guarantees and improved empirical performance in delayed reward environments.

Findings

Faster learning speed in inverted pendulum and lunar lander environments.

Improved final performance over standard PPO.

Theoretical proof of performance bounds and reward preservation.

Abstract

In this paper, we tackle the challenging problem of delayed rewards in reinforcement learning (RL). While Proximal Policy Optimization (PPO) has emerged as a leading Policy Gradient method, its performance can degrade under delayed rewards. We introduce two key enhancements to PPO: a hybrid policy architecture that combines an offline policy (trained on expert demonstrations) with an online PPO policy, and a reward shaping mechanism using Time Window Temporal Logic (TWTL). The hybrid architecture leverages offline data throughout training while maintaining PPO's theoretical guarantees. Building on the monotonic improvement framework of Trust Region Policy Optimization (TRPO), we prove that our approach ensures improvement over both the offline policy and previous iterations, with a bounded performance gap of $(2\varsigma\gamma\alpha^2)/(1-\gamma)^2$, where $\alpha$ is the mixing parameter, $\gamma$ is the discount factor, and $\varsigma$ bounds the expected advantage. Additionally, we prove that our TWTL-based reward shaping preserves the optimal policy of the original problem. TWTL enables formal translation of temporal objectives into immediate feedback signals that guide learning. We demonstrate the effectiveness of our approach through extensive experiments on an inverted pendulum and a lunar lander environments, showing improvements in both learning speed and final performance compared to standard PPO and offline-only approaches.