Categorical semantics of compositional reinforcement learning

TL;DR

This paper introduces a categorical framework for compositional reinforcement learning, enabling modular, interpretable, and safe task representations through the mathematical structure of category theory applied to MDPs.

Contribution

It develops a novel categorical semantics for RL, using pushout operations and zig-zag diagrams to formalize compositionality and unify safety and symmetry concepts.

Findings

Categorical semantics models compositionality in RL via pushouts.

Proves properties that unify safety and symmetry in MDPs.

Introduces zig-zag diagrams for practical compositional operations.

Abstract

Compositional knowledge representations in reinforcement learning (RL) facilitate modular, interpretable, and safe task specifications. However, generating compositional models requires the characterization of minimal assumptions for the robustness of the compositionality feature, especially in the case of functional decompositions. Using a categorical point of view, we develop a knowledge representation framework for a compositional theory of RL. Our approach relies on the theoretical study of the category MDP, whose objects are Markov decision processes (MDPs) acting as models of tasks. The categorical semantics models the compositionality of tasks through the application of pushout operations akin to combining puzzle pieces. As a practical application of these pushout operations, we introduce zig-zag diagrams that rely on the compositional guarantees engendered by the category MDP. We further prove that properties of the category MDP unify concepts, such as enforcing safety requirements and exploiting symmetries, generalizing previous abstraction theories for RL.