Integrating LTL Constraints into PPO for Safe Reinforcement Learning

TL;DR

This paper introduces PPO-LTL, a reinforcement learning framework that incorporates Linear Temporal Logic constraints to ensure safety, using automata-based monitoring and penalty signals to reduce violations while maintaining performance.

Contribution

It presents a novel method integrating LTL safety constraints into PPO, enabling systematic safety monitoring and violation reduction in reinforcement learning.

Findings

PPO-LTL significantly reduces safety violations in experiments.

Maintains competitive performance compared to state-of-the-art methods.

Effective integration of LTL constraints into policy optimization.

Abstract

This paper proposes Proximal Policy Optimization with Linear Temporal Logic Constraints (PPO-LTL), a framework that integrates safety constraints written in LTL into PPO for safe reinforcement learning. LTL constraints offer rigorous representations of complex safety requirements, such as regulations that broadly exist in robotics, enabling systematic monitoring of safety requirements. Violations against LTL constraints are monitored by limit-deterministic B\"uchi automata, and then translated by a logic-to-cost mechanism into penalty signals. The signals are further employed for guiding the policy optimization via the Lagrangian scheme. Extensive experiments on the Zones and CARLA environments show that our PPO-LTL can consistently reduce safety violations, while maintaining competitive performance, against the state-of-the-art methods. The code is at https://github.com/EVIEHub/PPO-LTL.