Hyperproperty-Constrained Secure Reinforcement Learning

TL;DR

This paper introduces a novel method for secure reinforcement learning that incorporates hyperproperty constraints expressed in HyperTWTL, demonstrating improved performance and scalability in robotic security applications.

Contribution

It proposes a new approach combining HyperTWTL constraints with Boltzmann softmax RL for security-aware policies in MDPs, addressing a gap in security-focused RL research.

Findings

Outperforms baseline RL algorithms in robotic security tasks

Demonstrates scalability and effectiveness in a pick-up and delivery case study

Validates approach through comparison with existing methods

Abstract

Hyperproperties for Time Window Temporal Logic (HyperTWTL) is a domain-specific formal specification language known for its effectiveness in compactly representing security, opacity, and concurrency properties for robotics applications. This paper focuses on HyperTWTL-constrained secure reinforcement learning (SecRL). Although temporal logic-constrained safe reinforcement learning (SRL) is an evolving research problem with several existing literature, there is a significant research gap in exploring security-aware reinforcement learning (RL) using hyperproperties. Given the dynamics of an agent as a Markov Decision Process (MDP) and opacity/security constraints formalized as HyperTWTL, we propose an approach for learning security-aware optimal policies using dynamic Boltzmann softmax RL while satisfying the HyperTWTL constraints. The effectiveness and scalability of our proposed approach are demonstrated using a pick-up and delivery robotic mission case study. We also compare our results with two other baseline RL algorithms, showing that our proposed method outperforms them.