Shielded Reinforcement Learning Under Dynamic Temporal Logic Constraints

TL;DR

This paper introduces a framework combining reinforcement learning and formal logic to ensure complex spatio-temporal constraints are satisfied during learning, enabling safer and more flexible robotic control.

Contribution

It extends safe RL by integrating STL specifications with control barrier functions, handling dynamic targets and complex temporal constraints in a model-free setting.

Findings

Framework successfully enforces STL constraints during learning

Effective in simulations with dynamic targets

Enhances safety and task compliance in RL

Abstract

Reinforcement Learning (RL) has shown promise in various robotics applications, yet its deployment on real systems is still limited due to safety and operational constraints. The safe RL field has gained considerable attention in recent years, which focuses on imposing safety constraints throughout the learning process. However, real systems often require more complex constraints than just safety, such as periodic recharging or time-bounded visits to specific regions. Imposing such spatio-temporal tasks during learning still remains a challenge. Signal Temporal Logic (STL) is a formal language for specifying temporal properties of real-valued signals and provides a way to express such complex tasks. In this paper, we propose a framework that leverages sequential control barrier functions and model-free RL to ensure that the given STL tasks are satisfied throughout the learning process. Our method extends beyond traditional safety constraints by enforcing rich STL specifications, which can involve visits to dynamic targets with unknown trajectories. We also demonstrate the effectiveness of our framework through various simulations.