Characterizing optimal hierarchical policy inference on graphs via non-equilibrium thermodynamics

TL;DR

This paper introduces a formalism for constructing hierarchical representations of Markov decision processes on graphs, connecting control, cognition, and thermodynamics to improve policy inference.

Contribution

It provides a novel theoretical framework for deriving hierarchical policies over all spatial resolutions using non-equilibrium thermodynamics principles.

Findings

Hierarchies correspond to a gradient flow between trajectory densities.

The formalism applies to discrete Markov decision processes on graphs.

It offers a normative approach to hierarchical policy inference.

Abstract

Hierarchies are of fundamental interest in both stochastic optimal control and biological control due to their facilitation of a range of desirable computational traits in a control algorithm and the possibility that they may form a core principle of sensorimotor and cognitive control systems. However, a theoretically justified construction of state-space hierarchies over all spatial resolutions and their evolution through a policy inference process remains elusive. Here, a formalism for deriving such normative representations of discrete Markov decision processes is introduced in the context of graphs. The resulting hierarchies correspond to a hierarchical policy inference algorithm approximating a discrete gradient flow between state-space trajectory densities generated by the prior and optimal policies.