Logic-informed reinforcement learning for cross-domain optimization of large-scale cyber-physical systems

TL;DR

This paper introduces logic-informed reinforcement learning (LIRL), a method that guarantees constraint satisfaction and improves optimization in large-scale cyber-physical systems by integrating logic-based projections into policy-gradient algorithms.

Contribution

LIRL is a novel approach that incorporates on-the-fly logic-based projections into reinforcement learning, ensuring feasibility and safety without reward penalty tuning.

Findings

LIRL outperforms existing methods in manufacturing, EV charging, and traffic control.

Achieves up to 44.33% reduction in combined makespan-energy.

Maintains zero constraint violations across all tested scenarios.

Abstract

Cyber-physical systems (CPS) require the joint optimization of discrete cyber actions and continuous physical parameters under stringent safety logic constraints. However, existing hierarchical approaches often compromise global optimality, whereas reinforcement learning (RL) in hybrid action spaces often relies on brittle reward penalties, masking, or shielding and struggles to guarantee constraint satisfaction. We present logic-informed reinforcement learning (LIRL), which equips standard policy-gradient algorithms with projection that maps a low-dimensional latent action onto the admissible hybrid manifold defined on-the-fly by first-order logic. This guarantees feasibility of every exploratory step without penalty tuning. Experimental evaluations have been conducted across multiple scenarios, including industrial manufacturing, electric vehicle charging stations, and traffic signal control, in all of which the proposed method outperforms existing hierarchical optimization approaches. Taking a robotic reducer assembly system in industrial manufacturing as an example, LIRL achieves a 36.47\% to 44.33\% reduction at most in the combined makespan-energy objective compared to conventional industrial hierarchical scheduling methods. Meanwhile, it consistently maintains zero constraint violations and significantly surpasses state-of-the-art hybrid-action reinforcement learning baselines. Thanks to its declarative logic-based constraint formulation, the framework can be seamlessly transferred to other domains such as smart transportation and smart grid, thereby paving the way for safe and real-time optimization in large-scale CPS.