Value Iteration is Optic Composition

TL;DR

This paper reveals that value iteration in dynamic programming can be understood as composition in a category of optics, providing a new mathematical perspective on reinforcement learning algorithms.

Contribution

It introduces a novel categorical framework for value iteration using optics, connecting dynamic programming with category theory.

Findings

Value improvement corresponds to optic composition.

Optimal value functions are limits of optic composition chains.

Illustrations include gridworld, inverted pendulum, and savings problem.

Abstract

Dynamic programming is a class of algorithms used to compute optimal control policies for Markov decision processes. Dynamic programming is ubiquitous in control theory, and is also the foundation of reinforcement learning. In this paper, we show that value improvement, one of the main steps of dynamic programming, can be naturally seen as composition in a category of optics, and intuitively, the optimal value function is the limit of a chain of optic compositions. We illustrate this with three classic examples: the gridworld, the inverted pendulum and the savings problem. This is a first step towards a complete account of reinforcement learning in terms of parametrised optics.