Inferring Causal Graph Temporal Logic Formulas to Expedite Reinforcement Learning in Temporally Extended Tasks

TL;DR

This paper introduces GTL-CIRL, a framework that combines causal graph temporal logic with reinforcement learning to improve learning speed and interpretability in complex, graph-structured decision tasks.

Contribution

It presents a novel closed-loop method that learns policies while mining causal temporal logic specifications, enhancing reward shaping and exploration efficiency.

Findings

Faster learning in gene and power network case studies

Clearer, verifiable behaviors compared to standard RL

Effective use of Gaussian Processes for parameter optimization

Abstract

Decision-making tasks often unfold on graphs with spatial-temporal dynamics. Black-box reinforcement learning often overlooks how local changes spread through network structure, limiting sample efficiency and interpretability. We present GTL-CIRL, a closed-loop framework that simultaneously learns policies and mines Causal Graph Temporal Logic (Causal GTL) specifications. The method shapes rewards with robustness, collects counterexamples when effects fail, and uses Gaussian Process (GP) driven Bayesian optimization to refine parameterized cause templates. The GP models capture spatial and temporal correlations in the system dynamics, enabling efficient exploration of complex parameter spaces. Case studies in gene and power networks show faster learning and clearer, verifiable behavior compared to standard RL baselines.