Non-Markovian policies occupancy measures

TL;DR

This paper proves that the occupancy measure of any non-Markovian policy in Reinforcement Learning can be replicated by a Markovian policy, enabling easier analysis and extension of theoretical results.

Contribution

It establishes that non-Markovian policies' occupancy measures can be generated by Markovian policies, simplifying theoretical analysis in RL.

Findings

Occupancy measures of non-Markovian policies are equivalent to those of Markovian policies.

The result simplifies proofs involving replay buffers and datasets in RL.

Applications to various RL scenarios demonstrate the utility of the main theorem.

Abstract

A central object of study in Reinforcement Learning (RL) is the Markovian policy, in which an agent's actions are chosen from a memoryless probability distribution, conditioned only on its current state. The family of Markovian policies is broad enough to be interesting, yet simple enough to be amenable to analysis. However, RL often involves more complex policies: ensembles of policies, policies over options, policies updated online, etc. Our main contribution is to prove that the occupancy measure of any non-Markovian policy, i.e., the distribution of transition samples collected with it, can be equivalently generated by a Markovian policy. This result allows theorems about the Markovian policy class to be directly extended to its non-Markovian counterpart, greatly simplifying proofs, in particular those involving replay buffers and datasets. We provide various examples of such applications to the field of Reinforcement Learning.