On Reinforcement Learning, Effect Handlers, and the State Monad

TL;DR

This paper explores using algebraic effects and handlers in functional programming to modularly implement reinforcement learning algorithms, separating decision-making logic from environment interactions, and discusses language and safety considerations.

Contribution

It demonstrates how algebraic effects and handlers can be used to implement reinforcement learning in a modular, language-agnostic way, extending the capabilities of the EFF language.

Findings

Reinforcement learning algorithms can be expressed as algebraic effect handlers.

Handlers enable separation of decision logic from environment, improving modularity.

The approach is compatible with lambda calculus with handlers, supporting language design considerations.

Abstract

We study the algebraic effects and handlers as a way to support decision-making abstractions in functional programs, whereas a user can ask a learning algorithm to resolve choices without implementing the underlying selection mechanism, and give a feedback by way of rewards. Differently from some recently proposed approach to the problem based on the selection monad [Abadi and Plotkin, LICS 2021], we express the underlying intelligence as a reinforcement learning algorithm implemented as a set of handlers for some of these algebraic operations, including those for choices and rewards. We show how we can in practice use algebraic operations and handlers -- as available in the programming language EFF -- to clearly separate the learning algorithm from its environment, thus allowing for a good level of modularity. We then show how the host language can be taken as a lambda-calculus with handlers, this way showing what the essential linguistic features are. We conclude by hinting at how type and effect systems could ensure safety properties, at the same time pointing at some directions for further work.