A Unified Theory of Compositionality, Modularity, and Interpretability in Markov Decision Processes

TL;DR

This paper introduces Option Kernel Bellman Equations (OKBEs), a novel framework for goal-oriented, interpretable, and compositional planning in high-dimensional Markov Decision Processes, emphasizing modularity over reward maximization.

Contribution

The paper proposes OKBEs that construct interpretable, compositional transition kernels for policies, enabling scalable, goal-based planning without relying solely on reward maximization.

Findings

STOKs enable compositional, modular, and interpretable policy representations.

High-dimensional STOKs can be efficiently factorized and computed.

OKBEs support verifiable long-horizon planning and intrinsic motivation.

Abstract

We introduce Option Kernel Bellman Equations (OKBEs) for a new reward-free Markov Decision Process. Rather than a value function, OKBEs directly construct and optimize a predictive map called a state-time option kernel (STOK) to maximize the probability of completing a goal while avoiding constraint violations. STOKs are compositional, modular, and interpretable initiation-to-termination transition kernels for policies in the Options Framework of Reinforcement Learning. This means: 1) STOKs can be composed using Chapman-Kolmogorov equations to make spatiotemporal predictions for multiple policies over long horizons, 2) high-dimensional STOKs can be represented and computed efficiently in a factorized and reconfigurable form, and 3) STOKs record the probabilities of semantically interpretable goal-success and constraint-violation events, needed for formal verification. Given a high-dimensional state-transition model for an intractable planning problem, we can decompose it with local STOKs and goal-conditioned policies that are aggregated into a factorized goal kernel, making it possible to forward-plan at the level of goals in high-dimensions to solve the problem. These properties lead to highly flexible agents that can rapidly synthesize meta-policies, reuse planning representations across many tasks, and justify goals using empowerment, an intrinsic motivation function. We argue that reward-maximization is in conflict with the properties of compositionality, modularity, and interpretability. Alternatively, OKBEs facilitate these properties to support verifiable long-horizon planning and intrinsic motivation that scales to dynamic high-dimensional world-models.