A Gentle Lecture Note on Filtrations in Reinforcement Learning

TL;DR

This paper explains the concept of filtrations in reinforcement learning, illustrating their role in modeling partial knowledge over time and arguing that the Markov property makes filtrations unnecessary for decision-making.

Contribution

It provides an accessible intuition for filtrations in RL and demonstrates that the Markov property renders their explicit use unnecessary for optimal decisions.

Findings

Filtrations represent partial knowledge in RL.

Markov property eliminates the need for filtrations.

Filtrations are not essential for decision-making in Markovian settings.

Abstract

This note aims to provide a basic intuition on the concept of filtrations as used in the context of reinforcement learning (RL). Filtrations are often used to formally define RL problems, yet their implications might not be eminent for those without a background in measure theory. Essentially, a filtration is a construct that captures partial knowledge up to time $t$, without revealing any future information that has already been simulated, yet not revealed to the decision-maker. We illustrate this with simple examples from the finance domain on both discrete and continuous outcome spaces. Furthermore, we show that the notion of filtration is not needed, as basing decisions solely on the current problem state (which is possible due to the Markovian property) suffices to eliminate future knowledge from the decision-making process.