Stratifying Reinforcement Learning with Signal Temporal Logic

TL;DR

This paper introduces a stratification-based semantics for Signal Temporal Logic, linking it to the geometry of deep reinforcement learning spaces and enabling new analysis and computational techniques.

Contribution

It establishes a novel theoretical framework connecting stratification theory with STL, applied to analyze DRL embeddings and improve computational methods.

Findings

Stratification theory can interpret STL formulas as space-time stratifications.

Numerical techniques applied to DRL embeddings reveal stratification structures.

Signatures proposed for uncovering stratification in high-dimensional embedding spaces.

Abstract

In this paper, we develop a stratification-based semantics for Signal Temporal Logic (STL) in which each atomic predicate is interpreted as a membership test in a stratified space. This perspective reveals a novel correspondence principle between stratification theory and STL, showing that most STL formulas can be viewed as inducing a stratification of space-time. The significance of this interpretation is twofold. First, it offers a fresh theoretical framework for analyzing the structure of the embedding space generated by deep reinforcement learning (DRL) and relates it to the geometry of the ambient decision space. Second, it provides a principled framework that both enables the reuse of existing high-dimensional analysis tools and motivates the creation of novel computational techniques. To ground the theory, we (1) illustrate the role of stratification theory in Minigrid games and (2) apply numerical techniques to the latent embeddings of a DRL agent playing such a game where the robustness of STL formulas is used as the reward. In the process, we propose computationally efficient signatures that, based on preliminary evidence, appear promising for uncovering the stratification structure of such embedding spaces.